I agree with vhdos and glisten, and will add you should talk with them about why you're upset and don't want to leave him with them. It sounds like you're frsutrated and mad at them but they don't know, or if they know you're mad since you don't let them see him anymore, they probably don't know why.
To me, you sound passive in this situation and it seems like you want to avoid the unpleasantness with the grandparents and just take him away, but this is backfiring since it's impacting his behavior. As his mom, you need to actively stick up for yourself and him strongly with them. As much as you dislike them you need to be strong on his behalf and for his benefit.
I understand where you're coming from because I strongly, strongly dislike my in-laws and do not speak with them unless absolutely necessary, but I keep my lips zipped when my kiddo is with them because I never want her to know mommy has had it UPTOHERE with her grandparents.
Tell them you really appreciated it when they called you before they took him places, and that for some reason they stopped doing this and it is unacceptable. Tell them you need them to start calling you again to ask you first if they can take him places, especially to their vacation house. You say they take the other grandchild with them (without asking the parents first?) - and unless you stated otherwise, they may assume you are OK with it. So tell them it's not!
If I were you I'd also tell them straight out you have learned they are trying to get him to call people other than his parents "mom" and "dad" and this is totally unacceptable and they need to stop it.
As for not trusting others around them (criminals, etc), I agree with the other suggestion to meet them somewhere outside of their home.
You are in control, so start off slowly and let them earn your trust back.
As for getting into stuff, running around, etc. I think that's pretty normal stuff for a 3 year old. I think you should reconsider your expectations of this type of behavior. He's curious and he's learning, so of course he wants to investigate things. yeah, he may make a mess or knock stuff over or whatever, but he's just exploring his world.Remove anything from his reach that is dangerous or breakable, and let him make a mess, run around, etc. 3 year olds don't sit quietly for hours on end. Unless he's deliberately or maliciously destructive, it sounds like he's just doing normal kid stuff.
Best wishes.