I think you should just refund the buyer, and get it off your shoulders. I can see what the buyer is saying...you said the phone is brand new, but that you played around with it. Technically, that could be considered "used" even if you only used it for 5 minutes. I myself am not THAT picky, so I wouldn't call it "used"...but apparently the buyer thinks a few minutes of use make it a used phone.
I don't think the buyer is trying to bully you. If anything, it sounds more like a case of buyer's remorse which sucks...I doubt the phone showed any signs of wear if you just pushed a few buttons or what not...but it's probably best to just refund to get rid of him and attempt to make him happy. Also, you already agreed to refund so from a moral standpoint you should follow through with it.
that's what it sounds like to me "buyer's remorse" you agreed to
a refund.. take care of that and be done with this unpleasantness...