I wear them together. I don't care what people think-- the velour is TDF. Soooo comfy and not sloppy looking.
I have a juicy suit in frenzy velour that all my girls constantly tease me about. Frenzy btw is a BRIGHT smurfy teal color. Yes, bright teal. And yes, I wear the jacket with the pants.
But, I get hit on more when I wear that outfit than any of my "sexy" clothes.
Its a win-win situation. It is a modest comfy outfit and guys love it.