Unfortunately this is something that some of us have to deal with. It really depends on where you work and what industry. There have been threads about this topic before and lots of people have said, "just wear what you want, who cares what people/your boss think!" However, that just isn't reality.
One day I was wearing CLs and a client made a comment about it. That was the last day I ever wore CLs to work. Even though fashion is something that I am passionate about, it is not worth it to draw too much attention at the office, or get yourself noticed for the wrong reasons.
Like others have said, you will have to either not wear your stuff to work, hide it, or find something that is under the radar. Have you considered a non-iconic Hermes bag? Most people are not going to have any idea what a garden party is (and it would be a great work tote). There are other under the radar bags and accessories too. You just need to find them. It is important not to upset people, but it is also important for you to be able to enjoy yourself (even if it is secretly!).