The terry runs smaller than the velour IMO. I would go with the larger of the two sizes...
It depends how you like them to fit (snug vs. a little looser). They do stretch out a little, I've found.
The small is great for everyday fit, as they aren't too baggy and are great for wearing all the time.
The size up is more baggy on the leg, but imo I think its more comfortable.
x