No, don't worry about what other people think of your jewelry. Wear what you like. 
Yellow gold looks best on my skin, but I also have a lot of silver, white gold, some platinum and some rose gold.
I like yellow gold best, but I do wear all the other metals also and I don't care if other people mistake my higher value white metals for silver. I actually love well worn, old silver with some patina. It's lovely.

Yellow gold looks best on my skin, but I also have a lot of silver, white gold, some platinum and some rose gold.
I like yellow gold best, but I do wear all the other metals also and I don't care if other people mistake my higher value white metals for silver. I actually love well worn, old silver with some patina. It's lovely.