I reorganize my closet at least once ever 3 months and get rid of anything I don't like or wear or want. I am now very careful in what I purchase and have kind of a uniform look going. I only buy white tops, from T shirts to very ornate lace.
Wow! You're even stricter than I am. I've recently moved to purchasing only neutral-colored clothing -- meaning black, white, any shade of grey, any shade of beige/tan/brown, olive green, and navy blue. I LOVE it. Almost everything in my closet can be mixed and matched, and I find it much easier to create stylish-looking outfits than I did when I just had a mishmash of colors.
I still have clothing in other colors from the days before I implemented this, but as it wears out, it is slowly being replaced with only neutrals. Then I buy shoes, bags, jewelry, socks, underwear, scarves, etc. in whatever colors I want to add some interest.
Anyway, sorry for the long digression from topic, it's just nice to know I'm not completely crazy.
~Ash