I've definitely experienced this.
My life really doesn't call for "getting dressed up" on a daily basis, but I do try and look pulled together- it makes me feel like I'm ready to start the day confidently. Even on days when I'm feeling lazy, I know I have things in my closet that are easy and not sloppy. Sweat pants outside the house/gym aren't allowed for me. Obviously you wear what you buy, and I choose to buy things that I feel fantastic in.
It does quite bother me when people comment on it, because I know they're looking for me to either talk myself down, or feel bad for them; neither of which I want to do. I get especially uncomfortable when I'm out with my boyfriend's friend's girlfriends- they all wear pajamas and/or sweats and talk about how "loaded" I am because I'm wearing skinnies, boots, and a cute top. I'm definitely not loaded- just know how to shop at Targét and find a good deal at
Nordstrom Rack :P
Just because I take the extra five seconds to button up a shirt or put on flats instead of ratty, old, fuzzy slippers shouldn't make me a target!