How do you respond to a guy who tells you that talking to you is wrong?

This guy I went out with a few times texted me out of a blue. I have nothing against him so we chatted a bit. The vibe seemed more friendly than sexual so I agreed to meet him for coffee.

I got a sense that he needed to talk to someone as he was telling me that he’s dating a woman who’s still married but considering a divorce.
She has two kids and still lives with her husband.

I listened and offered an ear for the future.
However, a few days later he told me that talking to me feels wrong because he’s in a relationship.

I didn't say anything but now I’m livid.
I’m single, have no interest in dating him meanwhile he’s dating a married woman with kids and yet talking to me is wrong?

I want to tell him he’s wrong but what do I say?

Do I tell him how I really feel?

How do you respond to a guy who tells you that talking to you is wrong?
Post Opinion