As a menswear designer I think their is nothing more chic then black worn with browns and navy. My outfit today highlights that.
VP - most men (and women) don't have your level of style to pull that off. LOL! You are a grade above! Just that small snap of your outfit was killer!!! I'm going to shove it under my hubby's nose! Understand, the rest of us are humans. 😊 So, we need to stick to some rules so our "basics" don't clash.
That being said, I absolutely don't think black goes with everything. Spring / summer fabrics and colors can be light & frothy. For those, black can seem harsh. Also, for some earth toned neutrals (beige, camel, brown), an earth toned accent, navy, a gold metallic, or jewel tone could be a better option.
To me this is solved with having several handbags in several colors. Play with them. Push your limits. Combine things in new ways. But, please don't default to a black bag every time. There's so much else out there!