Absolutely true. I guess the SAs try to gauge whether that'll happen, based on first impressions or whatever rapport they strike up with the first time customer. Not the best way to go about things, for sure, but in areas with high tourist traffic, that may be their only option. For that reason,
I like to try striking up a conversation with whatever SA greets me about the different leathers, different bag styles, Hermes history, whatever gets a conversation going that's not specifically about a Birkin. If I get a cold or curt answer when trying to talk about things like that, I know not to bother asking about a Birkin or anything else, for that matter!

Good luck, and enjoy your trip!