I do this pretty often. I think it's better as a casual look, though. For some examples, I have a sort of twig-brown cashmere grandpa cardigan that I love with a grey tshirt and black skinny jeans. I also like to wear brown leather accessories with black clothes. I have a bag, belt, and boots (I don't wear them all together, though!) in a warm, saddle brown shade that really pops against black. I would not wear a really dark brown with black, though. OTOH, I wear black and navy all the time, so it might work...