It happened to me years (eons) ago also at the Madison or used to be Fifth Ave store. I left my Amex number to my SA to have her charge my requests (a kelly and a couple of birkins) when they come in and have it sent to my apt in NYC or to my friends' houses as I was going to be overseas for a few months.
Voila! A few months later....
when I got my cc bill! I've got a TON, literally 15-20 bags shipped to me and my friend's addresses in NY/NJ area in a span of 3 months! I don't even like some of the bags shipped to me - like Yeoh bags (more than one, mind you), kelly ados, and bolides and kellys!!! I had to fly back to NYC to straighten it out. Got a car service to go through my friends' houses to pick up all the bags and then return probably half of them. The SA was let go soon after!!!
My current SA has my cc number but ALWAYS calls me up first to let me know if I wanted something before charging the item. Plus, I am there so often anyways... heh!