It really depends on your personal style, @swnsea21. A Sellier (outside stitch) Kelly will hold its shape, so all three leathers you mention would be fine. As I understand it, "Madame" is an outside stitch Birkin. So, that leaves box calf, chèvre and epsom to consider. Epsom would most likely be the easiest for you to get. It's a stamped leather and the least costly choice. Corner wear on Epsom means that corner color loss can fixed but the stamped pattern would be lost.
Of the remaining two leathers, Box Calf and Chèvre, my feeling is that Chèvre is more casual and Box Calf is more versatile. If your lifestyle is mostly casual, Chèvre is a great choice. If, instead, you want a day to night bag then I'd go with Box Calf. It's both casual and elegant, depending on how you style it. Best of all, it will get more and more beautiful with use. If you do choose Box Calf, don't be afraid of those inevitable scratches. Those scratches will blend and soften and help to create that marvelous patina.
One last thought on which size to choose. The mini trend will not be with us forever. The 28 size is not too small and not too big. I believe it's the size you would be happiest with in the long term.