I won't be much help, I like both.
The Neverfull seems more casual to me. The Totally is a bit more structured.
I wouldn't worry about other people thinking the Totally looks like a diaper bag. Other people dislike the Neverfull, too. They say it's too common, too unstructured. Other people don't like canvas bags because they think they look cheap. Other people don't like LV at all because they think people who carry LV are snobby. Other people...don't worry about the other people! You can't please them.
Either bag would make a great, everyday handbag to carry.
There is a big zippered pocket inside the NF where you can put your wallet, keys, etc. Safer. Or you could get a bag organizer that zips to put in the NF.
I LOVE my NF. But I live a very, very casual lifestyle, so it is a good fit for me.
But I would not turn down a Totally if someone gifted me with one. It's a really nice bag!