I'm partial to both leathers but for the boy bag, the lambskin just looks better. As it stands, the structure of the boy is quite masculine, I always feel that with the caviar on top, it looks a little bit too rugged. I know that corner wear one of the most common things I hear about the boy, but to be honest, I don't have any signs of corner wear on my lambskin boy. I do baby him a bit, but I also wear him to work probably two or three times a week and so far other than light marking under the flap, it still looks brand new.
As to bags I prefer in lamb or caviar. Light coloured bags, I avoid lambskin, like everyone else of tpf, I am terrified of colour transfer. But on anything smaller than a jumbo in darker colours, I will quite happily purchase in lamb or caviar. My one gripe with caviar as of late is that the washed caviar seems popular with Chanel and I only like the look of caviar when it is shiny (rigid).
I'm the same with doubts every time I purchase a bag, I always think of what ifs and could haves, but I am a firm believer that what you initially saw and fell in love with is what you should get.