You should do what you feel comfortable with, it's your choice and you can't be criticized for exercising that choice.
However since thoughts are being asked for and speaking generally and not referring to the OP as I obviously don't know them, my problem with "ethical" shopping is that many people are inconsistent in their thought process.
For instance there are well documented cases of animal cruelty in several "civilised" European countries eg bullfighting and up until recently stag and fox hunting, yet people will happily ignore that and buy products made in those countries whilst quoting the same reasons for not buying a bag made in china.
It was well reported on the conditions of the crocodile farms Hermes source their leathers from, I wonder how many of those women who say I'm not having a bag made in China would still buy a made in France Hermes in exotic leather? I'm assuming Hermes is made in France having never inspected one.
People are happy to buy a bag from a company such as Mulberry that is located in a part of England where the hunting & shooting fraternity are well represented. Some people will argue that hunting is not animal abuse at all others will say the exact opposite. Is it a case of some levels of animal abuse are OK when it suits? We might suggest skinning a dog alive is evil but people in China might find bullfighting equally evil.
As far as human rights are concerned whilst we don't have the same political regime as in China some might suggest our military involvement in other countries amounts to the human rights abuses of those citizens. These are not my own views I just provide them to point out that their are often two sides to every arguement.
For what it's worth I try to take a pragmatic view on such things and if Mulberry tell me their factories are well run and the workers treated fairly in China that is good enough for me until I have evidence that suggests otherwise. I know that it would be virtually impossible to live without goods made in china, think of all the electronic components in every day household items, so what is the point of me boycotting a particular bag when I have hundreds of Chinese manufactured items already, most of which I know nothing about because they don't have an obvious black label printed with Made in China attached.
Apart from anything else the laws that some countries have on country of origin labels means they are basically worthless, a bag with a made in Italy label could for the most part be made in China.