I also find that it's exceptionally easy to justify casual bags, but I don't think that's necessarily a bad thing. I think it makes more sense to have a couple of bags you know you'll be using day in and day out than to have a couple of bags you'll use maybe once a year. I've resorted to using cheaper clutches at TJ Maxx or Marshall's to fill those holes in my collection.
Sorry, I don't mean to sound like an enabler. I'm just saying that I don't think there's a practical need to whittle it down to one casual bag. It might make more sense to cut back in other bag categories (depending on one's needs)
I think that this is excellent advice - I am doing the same. IMO one's bag collection should mirror one's lifestyle. I don't go to a lot of formal events, so I don't invest in formal handbags. (I do love to look at them, though - but I am rarely tempted to buy, since I know that they will just sit on the shelf!)