No, absolutely not!
I prefer silver hardware and wear silver jewelry, but, my sunglasses have notable gold accents. I wanted silver accents but this style didn't come with silver accents. I used to wear a watch with mixed metals...stainless with yellow gold, which addressed this mixed metal concern nicely, although I no longer wear it.
Wear what you want and enjoy!!!