This is exactly what I have been doing and I feel so much better for it! My ideal collection would be two bags for 'better' or special occasions (already have my Chanel flap but want to add a YSL clutch in a nude/blush colour), two bags for everyday and a bag specifically for work/interviews. So no more than a total of five.
I would love to add a Chanel GST in black to my collection at some point but would have to sell a bag to do it. At one stage I had 30 bags (not all premier designer) and managed to get them down to 12, then eight, then five. I feel I can justify their cost more when I use them regularly and I just don't make enough use of each when I have too many.