I use my bag charm on my Neverfull but I hook to the ring on the side. I worry about marks from the metal on the vechetta. I think over time it will happen.
I've used charms on the handles for years and never had an issue with marks or rubbing. I don't put charms on my DE though since they might crack the handles.
I also wonder about the rubbing on the canvas, when walking and the swaying of metal charms against creating a rubbing action/friction on the canvas etc