You have to know that (I live in Paris) men carry Neverfull in Paris, so GO FOR IT !!! And, it's YOUR bag and YOUR life so don't worry about what other people think. You have to carry what YOU want and not what people want you to carry...
I've just bought a Speedy 40 (I'm a guy) and I LOVE this bag, so what's the problem, If you like the Neverfull, GO FOR IT !!!