It's not going to be a problem as long as you put something on the leather to repel soiling and staining. I've had a medium for about 4-5 years now and it is looks pristine and I use it in the summer months. If Chanel will seal the bag as I have read they can do, do it. If not, get something like luvinmybags cream protectant and apply before you even take it out for the first time. Lucky for me I followed my own advice and when an SA gave me her biz card and the ink transferred on the back pocket, it wiped right off with a little saliva. (I was out shopping and freaked and didn't have a baby wipe. Desperate times, desperate measures and this definitely qualified. I was about to cry when I saw the ink.)
I have a white Chanel but it's calf and not caviar. Wipe the bag down with a clean barley damp cloth or baby wipe every couple of times you wear it, let it dry completely before putting it away and general dirt shouldn't build up.