I don't take them off either. I usually add my charm on the opposite side. I think it makes for a beautiful more complete bag too..Plus I LOVE when my bag "jingles" or makes noise.
i read on here once upon a time where someone considers the hangtags to be like price tags, and always removes them...discards them too (or perhaps sells seperately)
i keep them all on, add fobs and scarves right along with the hangtags.