It's all about confidence. If you are self conscious about the bag you're carrying, people will pick up on your insecurity. So wear your LV as though it was just another gift from one of your many swains, the short froggy one, prince though he may be. Exude confidence and people will attribute that indefinable "It" to you and not your LV.
I agree with you. Confidence is the key unless a group of people are targeting you.
And for OP there is nothing wrong being middle aged and wearing monogram bags !