It depends! If I want to wear a certain bag and it's, for example, brown, I would try and work around that color. But if I don't care what I wear (just something to go to the mall with), I'd grab a bag that works with my casual outfit.
Ummm....I match my bags to the "event" of the day lol...eg. school= wearing batignolleH, and usually my clothes are just......(how can I put this....lol.....ummm....) ....everywhere lol. But I do tend to buy one main Winter jacket according to its colour so that it's easier to match and most of the time it's white, then I always wear jeans, and voila! lol I'm all set to go out in my region where Winter is ~70% of the year lol.