This is a good topic.
Birkins aside, I believe we should always dress well -- appropriate to our lifestyle and budget -- to present ourselves as best as we can, in or out of our homes.
When we allow ourselves to not care how we look
at all times, saying, "I'm just running to the store, this outfit is for comfort / no one will see me anyway", we fall into a way of thinking that there are some occasions deserving of our nice, fashionable clothes (and matching Birkin) and other occasions when we just couldn't care less about what we're wearing or how we look (rendering a Birkin a "non-match" in these instances).
Why not treat Birkins as any other practical accessory that completes an already polished look? How about a daily effort to be nicely dressed so it won't matter whether it's a Birkin or a cloth tote, this accessory will match what we're wearing?
It's like the attitude of many French and Italian women of-a-certain-age who never leave the house without well-coiffed hair, pressed clothes with clean lines, tasteful jewelry, nylons, and polished shoes regardless of age or dress size. It must take effort, but they always look great! And they probably don't give a second thought to their handbags.