I move from small to medium to large to huge all the time. My agenda is a bit of a problem, it's too big to fit in my small bags so those bags are just for short outings.
I lug a heap of stuff with me, and one of the best ways to cut back is makeup. For a huge bag, I might take a little extra with me. For medium bags, I take a slim Longchamp Le Pliage pouch and for small, I take only a lip liner, lip balm/gloss and lipstick. One of each. Another easy one is unnecessary items like sunnies for the evening (or long days at work when I leave before sunrise and return after sunset), large water bottle, scarf for warm days...
If there's one thing I hate, it's a bag that's stuffed full and the mess it inevitably turns into, so tend to go for a size larger that I actually need.