I used to be very matchy-matchy with my sterling silver jewelry, but I have forced myself to get over it, and I'm glad I did, because I was missing out on a lot of great bags with gold-colored hardware.
p.s.--What really helped me was acquiring some two-tone 18k/sterling silver David Yurman jewelry. That matches my bags no matter what color the hardware.
Cosmo so you think it doesn't matter much?