I always have some kind of charm on my bag, and so far I have never had a problem, so I wouldnt' worry too much. Don't hide the little guy away for fear of losing him! you want to show it off!!
Anyways, back on topic, I've never lost a charm off of my bag. If it is properly secured, I really don't see how you could lose it. I'm paranoid, so I always look at it, but I honestly don't think it would fall of unless it was really old and the clip had broken or something.I'll never lose my charm (get it?)
Bethie, just to put your mind at ease, just send that skull charm to me and I will take really good care of him for you!!