Back in the early 80s I worked in the art dept. (photo lab) for Reynolds Metals Co. here in VA. OK--first of all, it was THE 80s. Second, it was AN ART DEPARTMENT. Third--PHOTO LAB. This means a DARKROOM in the BASEMENT. We were all college students except of course for our many bosses. I used to get dragged into the office pretty regularily to be scolded for what I was wearing. I'll never forget the time I wore a pair of (really hot) leopard-skin-print jeans. My supervisor called me in the next day and said, "well, the reviews are rolling in!" You have to remember I'm in Virginia. This company was ultra-conservative, in fact the cafeteria was still segregated unti the mid-70s. It pissed me off at the time but now I think it's funny.
Your boss may have a bit of a letch for you, you know, which would explain why he is singling you out.