This guy is saying I'm not faithful to him, but we are not in a relationship?

I don't understand this. He and I are friends with benefits; we have never been in a relationship. And he even said we were not exclusive, that we were just casual. Now he is bringing up how I am not faithful to him because I told him I went on dates with other men. I don't get it. Am I missing something here? I did meet his mother, but he still has been saying we are just casual. He even said we were not sexual exclusive, but now he's talking about how I am not faithful?

Updates
8 mo
He even said it was ok if i dated other men but now he's saying i am not being faithful.
This guy is saying I'm not faithful to him, but we are not in a relationship?
Post Opinion