I don't have either but have seen both in person and yes it's a very noticeable difference. I could barely fit my phone in the nano. Way too small IMO.
I love the way the nano looks in photos, but when you compare the two sizes IRL the nano just doesn't feel as functional as the small. I was actually surprised to see how petite the small was, since in photos it always looked too big to me. But it's actually the perfect size - small enough ti be dainty and feminine but large enough to be functional.