I would go. This is one of the nicer outlets in the country (has nicer high-end stores). As mentioned before, try to go on a sunny day.
A lengthy drive (2.5 hours each way) one day Spring 2011 didn't stop me. I had seen these beautiful blue and coral bags in the local Ferragamo store Spring 2010. Went home with a pair of lovely coral Ribes but one year later, the bags were still on my mind. I finally returned to the store to ask "Where does your merchandise go after the season is over?" The SA said "Woodbury!"
So I made the trip and they had my bag! But spent hours debating blue or coral while I shopped. I also nabbed a second pair of my beloved coral Ribes (as I'd worn the first pair to the ground), got a cute pair of lime Jimmy Choo pumps ($200) and nice pair of black Theory ($80) shorts. Regret not buying some brown, practical Louboutin pumps ($300) at the Barney's outlet.
My outlet trip was on a sunny, warm spring weekday and there wasn't a crowd; very pleasant shopping. Maybe look up some of the nicer second-hand stores in Manhattan too, if you're into that thing; great finds in such a fashionable city. Hope you have a great trip either way! And congrats on the Mulberry bag, OS Alexas & Bays are my fav (I've been glued to the forum while I wait for my first Mulberry a Grainy Print black Bayswater to arrive tomorrow).