I wouldn't personally hang a charm of any sort on a vermis alma. It will scratch it terribly and over time could even cause it to peel, both where it is attached, and where it constantly rubs. If it does, that is not fixable. Since I also agree its too busy and the bag is much more stunning without it, I sure wouldn't risk marring and/or permanently ruining your beautiful alma.
Also, with any charm hanging from any LV bag, it's important to remember LV now uses just coated silver metal for their hardware. It can peel and lose its gold color over time with anything rubbing against it. (Just think about what happens to an LV key cles over time when someone actually uses it for keys). We've all seen pictures where very quickly the gold color has worn off and it's all scratched. So you'll have peeling of the actual vernis leather where it attaches, but it also will (not may) scratch and discolor that gold color hardware at the handle where it's attached. Then if you want to use the bag with or without the charm, you're going to have more wear, and perment wear, to just one side of the bag. I permanently scratched the hardware on my Trevi with a charm on it for less than a week. LV no longer uses brass. It's relatively cheap coated metal. I no longer use any bag charms on a bag for all these reasons.
Also, people who know LV will recognize this charm as not being LV, but close enough in style to it to look inauthentic and may wonder about the authenticity of your bag when they see it which is also a reason I would skip it.
The biggest reason though is damage to the bag (scratches) and handles (peeling) which is very likely to happen and which will permanently ruin the bag. I paid close to $200 trying to repair the beginnings of pinprick peeling on a vernis zippy and I used it less than 15 times when that started on the edges. I was careful with it as well. I see vernis items all the time in pictures here on the forum that mention "small white dots.". Those white dots are peeling. It's the vernis actually peeling where the glaze has worn. This happens normally even being careful. Hanging a metal charm that is constantly shifting and rubbing against the glaze and finish will not only leave permanent scratches, but could cause peeling that is irreparable. I would take it off as soon as possible.
My SA said she saw a vernis alma MM totally ruined by a tappage charm. I wouldn't take the risk.
think about it - if they do you saved at least £100!