I think if an invitation says 'casual' you can dress up if you want. If someone wrote 'casual' on a landmark birthday invitation to me, I would think they meant 'come as you are comfortable' and not to panic people into worrying about what they need to wear. Dress in your outfit as planned, IMO, people always love that someone made an effort regardless.
Someone can nearly always 'raise' a dress-code just not 'lower' it