This reminds me of a story.
I used to be a store tech for Verizon Wireless and I would do a test call from a phone a lot to make sure it was working. Most of the time I would just use the store line but if all the lines were tied up I would just use my phone.
So one night I'm chilling and playing Eve at about 9:30 PM or so and my phone rings from some random number. I pick it up and immediately someone yells, "WHO THE HELL IS THIS?!?"
So, I don't take this kind of crap off of people and I inform his happy ass that he was the one who called me so he can damn well tell me just who the hell he is first. That is how polite society works lunkhead.
So he goes on and on about how he found my number on her phone and how she is cheating on him with me and how he is going to do so so to me, etc. Whole time I'm thinking man, you got some serious problems in your life if you are snooping through a phone and freaking out over one call from a random number.
Fortunately I was able to convince him I was a tech at Verizon, I just fixed her phone that day, and that he needed to apologize for being such an ass.
Point of the story is if you cringe and whine you are seen as weak and untrustworthy by this person and confirm everything he is thinking. You need to be strong, forceful, calm, and confident. That gets people like that to back off most of the time. Stay safe OP