Like some of the Posters have stated, I attach my Charms on my ball chain of the hangtag, too. Yes, I worry that the ball chain may break so I check it ALL the time! LOL
As an alternative, you can purchase carabiners or the like (from hardware stores - near keys, sporting good stores, etc) so you can attach it to your purse's O ring and then attach your Charm (or keyfob) to the carabiner. This also minimizes scratches to your hardware when you try and attach your keyfobs, too.