Most Helpful Guy
Try and see a couples therapist before you give up. If he's unwilling then there's not much more you can do and at that stage, it may be best to remove him from your life. You deserve to have happiness and love. And your son will be fine as long as his parents love him.
Most Helpful Girl
Honestly, he sounds emotionally abusive. He only brings that up when he's angry? Probably just to hurt you. He's talking to someone else? He doesn't respect you. You don't feel good enough? He isn't taking the time to appreciate you (or maybe communication is lacking but that's doubtful).
I think you should have some self respect and leave. I tried to stay with my partner for the sole purpose of our child and it emotionally destroyed me. He didn't truly love me and it wore away at me say after day for months until I became so broken I forgot who I even was. That is not the sort of relationship I want to model for my son and I doubt it's what you want for your sort either. I know it's very, very hard and it's easier said than done but I do believe you need to easier this for your own good and your child's. At least see a counselor if he is willing. You should not have to be hurt line this, you and your baby deserve better, I promise.