I understand your reservations about the Tory Burch. I already have a plain black pair, so I like the gold. The leather is also unbelievably soft on the TB, but the elastic backing makes it a bit difficult to size. Definitely get your normal size, if not one size smaller. I think they also have some styles with a leather medallion which is a little more understated.
I also have a pair of Delmans that are quite nice. I would look at them as well. There is a plain pair at Bergdorf Goodman if you are in Manhattan.
I also have a pair of Jeffrey Campbell love flats, but didn't mention them because you specifically asked for a designer pair.
I would also check out Marc by Marc Jacobs, although I can't think of any styles off the top of my head.