Hi! I have two mediums (previously called small). One is black and the other is green. I love both of them and cannot use the small (previously called mini). I find it too small inside, especially with the divider. I really hate that divider in the Coco Handle because it takes up so much usable space. The small/mini is also "cute", and cute is not really my look. I think the medium/small looks more elegant which is more me, so it depends on what look you want.
On your color question...
If you love the caramel color and can use that color, then go for it. I know it's an It color now, but I have a caramel caviar bag that I rarely ever use. I bought it thinking it was such a neutral and I can use it year round. For a neutral, I use my beige claire much more. That said, I think this season the medium only comes in caramel/brown I think is the official color name), navy and black. Of the three, for me, I would get black. I honestly find it very hard to use my caramel bag, but I wear a lot of black in the winter, and for summer, I wear a lot of white and bright colors so caramel just does not work in the spring/summer for me. Think about what 5 outfits you would wear with that bag and if it fits your wardrobe. If it does and you love it, for sure, I would get the caramel in medium. Medium is a perfect size to me.
I hope that helps with your decision.