I've never actually been afraid to use my bags because that's what I bought them for.
However, I HAVE just not carried them in certain situations where I don't want to draw attention to my bag. In those types, I'll either carry my Epi Jasmin or my Damier Speedy.
I agree! I do that a lot, part of the reason why I've been buying more epi and damier in recent years too! Sometimes though, in like bad storms I may just grab a pochette (we get some really windy rainy storms where I live on the east coast) I usually don't get too worried when things get a little wet, but when rain is as heavy as a shower it's just a pain and I don't exactly want any of my bags to get completely soaked....