I believe those are mold spots. They are permanent. If it were mine I would dunk (bathe bag in a tub of warm soapy water) and put vinegar in the rinse water to make sure all the mold is killed.
ETA: Read through this thread so you can see the process and what conditioners and tricks are recommended.
Oh mold! Yucks! Thank you so much. I definitely will read through. Thank you!