If you aren't 100% gaga over this bag it doesn't matter what anyone else feels, you have to feel comfortable wearing it. If you love it, keep it. If not, it's back to the store and surf TPF for ideas of what bags you are attracted to. You might like the Rebecca Minkoff bags or Michael Kors. - Just naming a couple that fit in with casual attire!
What ever you buy, love!! I'm about to buy a bag that will cost me a fortune but I have to have it because I LOVE it!!!!