It depends on the bag for me. Most of mine have silver HW (I feel that's more casual), but I have one gray bag with gold HW and I think the color combo is just dreamy
I do like both gold and silver hardware, but I feel like it really depends on the style and/or color of the bag. I think that gold hardware has more of a classic vibe, whereas silver (my current favorite at the moment) is more rock and roll/edgy