After reading countless votes for caviar or lambskin, I finally have enough personal knowledge to weigh-in on this! I have a lambskin in black. I liked the way the black bag looks in lambskin a lot better. I will say that the lighter-colored bags were absolutely beautiful in caviar. I wanted a bag that was beautiful and worth the investment. I thought the lambskin was prettier, and heard the caviar was more durable.
I struggled with the caviar/lambskin decision for a while because I'm (a) clumsy, (b) morally opposed to a high-maintenance handbag unless it's a once-a-year evening clutch, and (c) planning to keep this bag so long my grandkids might see it (and I don't even have kids yet). Like I said, I liked the lambskin better, but I was super scared by the forum entries that said it was hard to maintain.
I've only had the bag for a few days now, but I don't think the lambskin is as flimsy as everyone says (just knocked on wood as I wrote this!). I read entries where people said it scratched the moment they took it out of the store, or that the store versions were scratched up. I should say that I don't have very long nails, but I wear rings and I am not super worried. I wouldn't toss it on some gravel and sit on it, but I'd say if you like lambskin better, go for it. A lambskin is the way that Chanel designed it (I think--please correct me if I'm wrong), and that lady sure knew what she was doing. Also, I'd like to put in my vote for medium too. It has the pretty red double flap inside. It's actually really useful to make sure your bag is secure and closed, which is important on the busy NY streets.
I love this bag-- it makes every outfit look great. Have fun with your purchase!