I have one of each size and use the Small the most. I don't like the Medium because the walls are too high (for my needs) and the one I have is a soft material so it collapes in on itself when I put things in the outer pockets.
If I use a bigger bag, I just drop the Small inside the Large. That way, it gives me more space/pockets, the Small doesn't tip over and I don't have to change out/re-arrange my stuff. But if I could only have one, I'd pick the Small as the most versatile.