It's also just what you notice. If you tend to buy smalls you're going to notice when there aren't many. If you tend to buy larges you'll notice when there aren't any of those. Think of it this way---when you go to the grocery store you may always think you choose the longer/slower line, but that's because you'll always remember when the line took forever. It's rare for a person to remember all of the times that the line went quick. So if you relate that to shopping, you'll remember the times that your specific size was out, but it's not going to stick out in your mind when you went to a shop and they had an abundance of your size.
I actually read a book a while back called The Drunkard's Walk---it's all about randomness in our world/lives. But one of the things that really stuck out to me is how you recognize one thing (i.e. never having your size at a store), simply because it relates to you, when in reality it may not "always" be that way.