No. You can mix any and all neutrals at will. If you are worried about it, add one other black, and one other brown, element to the outfit.
Or something that has both, like a string of beads that contains both black ones and brown ones, a peep of cami with a black and brown print, etc.