Most Helpful Guy
I would put it bluntly to him.
It is possible that he is as confused as you, so it is best you don't ignore him - that will show him his worst fears of you having other men are real.
Instead, I would tell him the truth - you do have other guys numbers on your phone, but he is your one and only. And you want to get back together, then that's great for the both of you; if not, he can go his way, but there will be no hugging or kissing and no contact.
If he will be so stubborn as to not believe you that he is your one and only, then he has trust issues and realky shouldn't be making such a big deal of it. You told him the truth - believe it or leave.
If he chooses to leave, you know what time of man he is in the future - that means no guy friends for you because "he will assume the worst".
Be careful of such possessive men. You will have to tell him that either you have guy friends that are no more than friends, or you do not enter the relationship with him, because frankly, one dude isn't worth the thousands of connections you can gather from having guy friends.
Let me know what you think.