In reality, Frances tried to get custody but lost out to the father, John Spencer, 8th Earl Spencer
I can remember reading somewhere ages ago that Frances' own mother testified against her in the custody hearing. I'm sure that a child's eye, it would feel like abandonment.