I like and own both. It doesn't bother me to mix hardware-it's not like you are wearing the wallet on the outside of your bag. I usually change my handbag 2-3 times a week and I don't have time to switch out wallets that often too. As far as I am concerned, there is no right or wrong answer-it's all about what you like and what bothers you. Some women will not wear white metal jewelry with a handbag that has brass hardware. IF you NEED to match, then go for it!
I don't try to match mine, either by hardware or color, I just use what I like. I'm a gold hardware gal and rarely buy a bag with silver hardware, but I do have quite a few accessories and wallets with silver hardware because I don't mind it there.
Oh please! I would go crazy if I tried to always match everything! Mixing metals is big right now anyway! Some Coach bags even have two different colors of hardware on the same bag. The last thing I worry about is if the hardware on my wallet matches my bag. Usually I don't even worry about matching my jewelry to my bag. I focus on the big picture and looking chic and pulled together