White gold usually looks a bit dingy and yellow compared to silver, just like platinum always looks a little grey by comparison.
Are most people going to notice? No.
Most people assume that a solid shape is silver, anything with stones is gold, and a wedding ring is platinum because that is what they are used to seeing.
As for the white gold being more popular than yellow? I suspect that's because cheap yellow gold is obviously of lesser quality than the high carat stuff, but 10K white gold and 18K white gold look the same until the rhodium wears through.