You could definitely remove it without damaging the charm, however, the link you open (there are several links between the chain strap and the logo charm) might get scratched or damaged in the process. The links are pretty thick so they might be hard to pry open, even with heavy jewelry pliers. Still, it can easily be done if you're willing to risk a "tweaked" link.
P.S. I agree that it can also be tucked inside. It might slip out every now and then but for the most part I think it would stay hidden.