Good question. It seems that many prefer either one wallet that must fit every bag or coordinate their wallets with their bags in colour, brand and style, using similar size wallets with similar bags for whatever they're doing. However, I like different size bags for doing different things so I would rather have a few neutral coloured wallets in different sizes.
I have roughly 4 sizes of wallet. Ag ATM I use flat huge Hermes which doubles up as a clutch, a large
Gucci which could also be an evening clutch (phone included) a compact Gucci which would only hold cards/notes and a few coins and and a mini Gucci that really would only hold cards and notes (the compact and the mini are exactly the same on the outside except for the sizes).
Gucci have several styles of compact wallets and even zip-up card holders. Card holders are fine if you don't need to carry cash but normally, the smaller you go with a wallet the more need to carry a separate change purse which may defeat the purpose of 'downsizing'.
Hermes also have compact/combined styles of their popular style wallets.
I don't know about all other brands but my guess is that there are smaller wallets available and around but they won't be 'front page' or front of store but kept in draws.
Some pouches, small and mini bags have in-built card slots and an attached purse like my Delvaux Hannelore going away for separate wallets and some larger wallets convert into bags with an added strap. I do think designers do think of these things but as you say it can mean additional time spent transferring thing from one thing to another.
