FYI, I learned this trick from working in retail.
If we came across money that we weren't sure was real or not, we'd rub it against white paper. If it transfers ink it's real.
So even paper can cause color transfer, like shopping bags from stores, money, boxes books, coffee cups, pad of paper or another handbag.
I see a lot of people holding their bags in the same hand as a paper shopping bag from some store or while drinking coffee. You might not have gotten the color transfer from your clothes, it could have been from something unexpected like while you were trying on clothes, accidentally brushed against another shopper or a rack of clothes, handling money or a gift box or shopping bag. Try not to take your light bags shopping when you will be picking a lot of things up in your hands, constantly touching money or picking up and putting your bag down a lot, there are far too many things that could cause color transfer that you wouldn't even think could cause it when doing something like shopping or running errands.