Aww, I agree with you completely and don't think anyone would ever boo you for this!
I think, well at least for me, it's important to keep it all in perspective. To have a designer bag is a luxury in itself, but to have time or energy to waste worrying about the bag (is it the right one? should I have gotten another color?) is the ultimate luxury.
When I see people on the forum fret, argue and even go behind each others backs to get a coveted purse, it all seems cuh-razy to me. I picture the CEOs of these companies throwing out bits of "limited edition" leather to the masses and watching us go crazy while they wipe their behind with $100 bills.
I adore gorgeous bags, but if they ever cause me to, as you say, get financially nervous, then it is time for me to sit back and live vicariously through others who have pretty purses. At the end of the day, when I think of the people who influenced me most in life, none of them had designer purses. It truly is just a bag.