I think the issue may be the wording. For a boss, or anyone in a professional setting, I'd go with 'I'd rather not discuss it' instead of 'none of your business'. And at work I'd never say 'don't touch my bag and I won't touch you'. That would get me a conversation with HR - the difference in touching an item and touching a person. Yes, boundaries need to set, but in a way that won't reflect badly on the speaker.
Well said!