I prefer silver hardware and wear silver jewelry, but, my sunglasses have notable gold accents. I wanted silver accents but this style didn't come with silver accents. I used to wear a watch with mixed metals...stainless with yellow gold, which addressed this mixed metal concern nicely, although I no longer wear it.
I think it is fine to mix. I usually wear silver jewelry and right now I am carrying my LMM in chocolate with gold hardware. I think it is fine. I don't think the hardware on your bag has to match your other metals.