If you're smaller in the chest area get an XS.I'm 5'3, a size 27 in pants and usually a 2 or 4. The Karta dress that i bought i believe is a small. I got it at Nordstrom Rack though, so i never tried on an XS. It's a tiny bit loose but it's very comfortable and i think it fits pretty well. I have a bigger chest than most XS people though, so a smaller size might have been too tight.
not sure if that helps at all.. but i would go for a small in that dress (if you can find it somewhere else) cause the medium might be TOO baggy.