I think the key is how well a bag integrates into your existing wardrobe.
I've been of two minds on this with my own collection. I sold a Fendi B-Bag because while I'd thought it would add vintage charm to my outfit, instead it just looked dated. I also have TWO multicolore speedy bags from LV. I love all the color and having a little piece of LV history, but sometimes it's hard to incorporate them into my outfits because they reflect a style that doesn't really match what I wear now. I love the Dior Detective, but I've been leery about buying one and having the same issue.
That said, I have a Celine luggage bag, and I never think twice about wearing it, because it still perfectly matches my style. Obviously that's a more recent design, but my taste hasn't outgrown it yet.
I love the Spy, and I would buy one in a second.