Same. I shopped with 3 SA's multiple consecutive times before deciding on one (who I've now been with for ~2 years). I also bought from other SA's occasionally (whoever was available) before I decided I wanted to be offered a bag from SF (I had relied exclusively on going to Paris for bags previously). I shopped with the same SA every time until something turned me off. For example, one SA was very pushy to get me to buy things that I wasn't interested in. I felt the SA suggested that I needed to spend a lot more "than usual" (1:1, 2:1) to be offered a bag (personal suspicion). I felt like they weren't taking my wishlist seriously at all. The second SA I worked with (maybe having a bad day?) exasperatedly exclaimed once in front of me, along the line of "every SA has helped you before!" - I'm not sure what that was trying to get at but it made me feel bad. I spent a couple thousand with every SA (not hard at Hermès as I'm sure you all know!) before deciding that we didn't jive well. Those moments made me feel uncomfortable personally (and maybe you would never experience those moments working with the same SA - it could be entirely circumstantial and personal to me). The Hermès "game" can be frustrating, so I want a relationship with someone who I get along with and goes out of their way to help me find the things I want, because I only want to buy the things I love and not to spend just to be offered a quota bag. I found an SA I click with now. Even though I feel like the SF store stock seems really lacking right now (for ready to wear & home goods), I'm waiting patiently for my next bag! I've been offered 3 quota bags (B30GHW, K28 Sellier PHW, C18 Ostrich). I bought the B30 and a harder-to-get black GHW Kelly Ado.
Honestly, I have no interaction with any other SA's especially now that appointments are timed, 1-1, and made in advance. I don't know if it affects me in any way if the other SA can have any say in whether or not I am offered a bag - but I doubt it (unless the person is a manager of some sort maybe?)
Maybe in retrospect I would have bought smaller ticket items when I'm testing out the relationship with a new SA to not feel the sunk cost fallacy so much. I think you should look for people who are excited regardless of how much you spend and spend the time to work with you and chat with you. The other important thing is how quickly someone gets back to you and their preferred method of communication. SA's will obviously try to communicate with you the way you prefer, but they often respond faster in their own preferred communication. Email sometimes might take a couple of days to respond but a text you could get a response the day-of. Some SA's are more busy because they have more regular clientele or are worse at responding on time.
DM me if you want to talk more

I also think your SA definitely warms up to you more the more time and money you spend with them because everyone wants a regular client!