I think if you worry about wearing a bag, you probably spent more on it than you could afford. I am not saying you should be careless, but you should be equally as comfortable with all of your bags. If you're not comfortable, IMO, you probably spent too much on it.
I don't think this is true for everyone. Some bags aren't easily replaceable, some are irreplaceable (e.g., seasonal colors/styles). I'm persnickety. With the quilted bags, I'm worried about deflating which I abhor. Certain defects cannot be reversed.