Gilroy could probably be done on the way to or from Santa Cruz should you want to go to that outlet. Gilroy is about an hour east of from Santa Cruz (traffic dependent of course). Depending on the time of year though, it can get a little stinky thanks to all the garlic. I find that a good stink though!
Also if you're on your way to Santa Cruz and like seals, I suggest Ano Nuevo state reserve (
http://www.parks.ca.gov/?page_id=523). I've been there once before and got to walk among the seals on a tour. It was amazing.
Napa & Sonoma would require a different day from Santa Cruz/Gilroy. Even though Napa & Sonoma are right next to each other, I found the regions quite different. When I went (which was a few years ago, so I'm sure things have changed), Sonoma wasn't quite as touristy I didn't think.
I'm not sure what kind of driving conditions you're used to, but I always found the idea of driving in SF very intimidating and attempt to avoid it. It might be worth just doing a one day rental here or there if you're going to spend most of your time in the city. Their public transit is pretty good.