I have actually thought the same thing before, though I never have used one... I've seen some great looking ones & seems one could get a much better deal- why are men's items always better priced???
I don't use one, but I have seen some men's wallets that look really nice (and like msbuttercup said, they can be a lot cheaper too!), so if I found the right one, I would definately consider it! Don't see anything "wrong" with it at all!
I use a unisex wallet from LV. It looks more like a men's wallet but I like the way it holds my bills flat. You can use whatever you want. I don't think it's a big deal.
The wallet I have (which I absolutely adore) is supposedly a "men's" wallet. I don't care. Designating one thing only for men or only for women is silly anyway. John 5 has this same wallet.