Where I work, there are very few women (architecture firm). It's a small company, and all the girls carry designer bags. I don't think the guys even notice. So I've begun carrying my Chanels although I was EXTREMELY reluctant to do so (I used to strictly carry my nonlogo Longchamp to work).
I would never carry a designer bag to work if I received flak for it. Something as important as your career is not worth the "Well, why shouldn't I if I want to" mentality. Yeah, it's not right for people to judge you, but they do so anyway... unless you don't carry that bag!