I really understand your dilemma because the exact same thing happened to me!
The bag is definitely STUNNING, almost looks like a piece of art or jewelry. I bought mine last Fall, the very minute I saw it, I was so in love. And then, same as you, started doubting it a couple days later. I am no stranger to metallics for casual use...my two favorite and most used shoes are metallic (one gold, one dark silver), but there was just SOMETHING about this bag that didn't work with my casual outfits. I think it may have been the size. So I went back to Dior in the hope they'd tell me that it also came in the small size, which I think would work better with casual looks AND be a better size for evening looks! Sadly, it doesn't, and I didn't think the WOC version did the color justice. So, I ended up exchanging it for the small onyx black micro-cannage diorama (the gunmetal one), and have been wearing that non-stop, day and night.
I still dream about that champagne color though and have asked my SA to email me if they ever make it in the small size. She did text me recently to alert me about another medium size gold diorama that is coming out, this one may be easier to style casually, though doesn't have the same WOW factor IMO. Here is a pic in case you are interested.