Guys am I transphobic just bc I refused to date a man who has a past of dating a Trans woman?

English isn’t my first language, sorry if I make mistakes.
I moved to Seattle about a year ago. I met a guy and we were seeing each other, everything was fine. Then I saw an old photo on his social media from a couple of years ago where he was very close with a trans woman. When we talked about past relationships, I asked him about it. He said they dated before but broke up.
After that, I honestly lost attraction. I didn’t mean to be rude, but I got quiet. When the bill came, he wanted to pay, but since I didn’t plan to see him again, I paid and went home. The next day he asked what was wrong. I told him I’m only attracted to completely straight men and that I can’t be comfortable with someone who dated a trans woman. I said I’m not judging him, it’s just how I feel. He said he’s not gay and that his ex had bottom surgery, but that didn’t really change how I feel. Then he got angry and threatened to report me at work for being transphobic and even said he would sue me. I don’t think I’m transphobic. Is this really how things work in the U. S?

Guys am I transphobic just bc I refused to date a man who has a past of dating a Trans woman?
Post Opinion