Medium. To me, medium is equivalent to the Small Longchamp Le Pliage tote or a LV Speedy 25/30 (edit: Speedy 30 is getting on the "large" side, imo). These are enough to carry my wallet, keys, "emergency" kit (explained in next paragraph), a lip gloss, and cellphone. Plus a little more room to carry extra items, if needed.
Small bags I like, but modern keys and cell phones make it hard for me to minimize. I also like to take an "emergency" kit that includes pain relievers (I get headaches easily), feminine products, and a lip balm. I used to not have this "emergency" kit in my purse, until I realized I started wasting money and buying stuff because I needed these things out of the blue when I was carrying a small bag. So these I only use for evenings or quick errands when I'm going on foot.
Large bags I like, too, and I use them everyday because I work full-time. However, I only use these for work and travel; they are too bulky to walk around in on a daily basis. I'll sometimes supplement a small or medium bag with these because I don't like digging around my purse to find what I need.
Also, I prefer to have my larger bags as totes and using coated canvas and or nylon. I hate carrying heavy bags, and if I'm going to schlep a lot, I'd rather not have a bag that adds more weight to the stuff I'm carrying. Small bags I'd rather have in leather.