MM--I don't think it looks weird, but it sure can get a bit unwieldy. When my DD was in the infant stage, I couldn't wield a handbag at all--I was still learning the ropes as a rookie mom! I just stuck my cellphone and wallet into the diaper bag and that was pretty much it. Even now, DD is a toddler, I only use my Birkin when I'm alone or DH is around to wheel the stroller. I'm at the stage where we still make her meals for her (we're a bit anal) so I still need the diaper bag to house her meal, diapers, snacks, restaurant toys, etc.
In short, I say go for it if you can handle it, but if you need a separate purse, I'd pick a shoulder/messenger style.
And I am super-casual when I'm w/DD--jeans, sweater, newsboy cap (same reasons as Gazoo--sometimes there's just not enough time to do the hair), clogs.