The track really varies-there are tons of people in shorts and tshirts and fewer dressed up. In my experience you dont see the dressed up with a big hat Kentucky Derby look on a regular day of racing. I would dress for the event itself (work party? social? time of day?) and wear something that will be comfortable indoors and out.
It depends on what track you are going to, and what section you are sitting in. People get much more dressed up at Churchill Downs than any other track I've ever been to (which I think is a good thing). People in the boxes will be much more dressed up than those in the grandstands. I would wear a nice sundress.
That is a tradition for that track. My DH is from there and we try to go to the Derby every year or every other year.