I have the older M3xxx Taiga leather passport cover, and I think it's totally worth it having a nice passport cover. Especially if you keep everything travel related together.
Mine holds my passport, WHO international certificates of vaccination, business cards, relevant membership cards. And I like the fact that I can throw this "package" into a safe and be done with it.
It's true that most officers working for federal authorities won't touch passport covers/ wallets, but that is really no problem. Since most of the time you're in a line, simply remove passport while waiting. And if there is no line, there's no delay caused by taking out the passport in what is like 10 seconds, maybe?
If an (immigration) officer makes a fuss about it anyway, they're mostly having a bad day. I always smile and say: "I'm sorry, officer".
Kind regards,
Oliver