You should definitely go for it. I have a bone white caviar M/L that I adore and use a lot. It's several years old but looks pristine, and any dirt will wipe right off. I had to use an eraser to clean some of the dirt I got on it, but I got it all off.
I also have a stark white caviar Mini that I use all summer. I got color transfer from my navy blue cotton sun dress but I was able to clean it with Apple cleaner.
Honestly, as long as it's caviar, do not afraid of whites!!!!!!