Yes you can!! Case in point right here, LOL. 99% of my bags are black or choco brown and with the exception of a few (such as my Felix Rey straw tote), I carry them year 'round.
You can get away with that in this part of the country. When I've been in southern climes (s. Florida, for example) and am wearing black I feel a little 'heavy.'