I definitely recommend hitting BOTH boutiques while you are here, if at all possible! Madison Ave is huge, and quite a sight to behold, especially at Christmastime when it's a total madhouse. The service there is often less than warm, though, so just be prepared.
Wall Street (actually the location is 15 Broad St near Exchange Place just so you can find it easily) is staffed by absolutely wonderful SAs, and though they are much smaller than Madison, it's often considered by regular shoppers to be a "better" store. No attitude, never overwhelmingly crowded, and they do have & display lovely items (including Birkins) on a regular basis. Try to get there early in the day if you are on a Birkin quest though--they open at 8:30AM. Madison opens at 10AM.