A client of the house is expected to show both taste and discernment, and even if it fit, but you hated it, for example felt fat in it or it reminded you of an unfortunate dress from the 1980s, it’s perfectly okay to refuse politely. The reasons you give will help your SA figure out items that may interest you in the future
At the opposite end of the decision spectrum, and not the case here would be the following scenario. If the dress were something you absolutely could not live without, and if there are no other sizes, you could always ask for a consult with the in house tailor. Chanel, for example, has multiple seams with two inches of fabric on each side, and can often be let out or taken in. Dior usually has no seam allowance, so no joy. But, at times, both chanel and dior head tailors can authorize additional fabric to be added. Hermes, on the other hand, may have structural issues that preclude even a shorten sleeve; and Brunello can alter most garments except the lower legs of pants. . . My point is if one wanted to make something work, there is no harm in asking. (A tailor will always distinguish between what can be done and what should not be done)