I've never had any problems with this, but I also treat my wristlets before using them. I think that you would most likely only have a problem possibly with light coated canvas, like the tattersall, or with patent..and only if you stored this type of wristlet/wallet in the bag unprotected for a long period. They now include white dust bags with the coated tattersall and patent bags, and this is the reason...prolonged storage against the fabric will cause transer on to the bags.
I even got a brown dust bag with my sunflower Bridgit and burnt orange satchel, so I assume they don't perceive any issues with these colored leathers either.
I think your problem will most likely be dirt, so that's why I usually condition and/or spray mine before using. It provides a barrier and makes stuff a little easier to remove with less harsh substances. I try to avoid leather cleaners in general, and on the fabrics I usually use dilute soap solution or Oxy Spray Away...both work great usually.