One day he wants me the next he doesn't. From a guy's point of view, what's going on in his head?

This guy I've known for 15 years since we were 17...we had sex back then and are having sex now. I am having an affair with him. I live with the father of my one year old daughter. The guy I'm having an affair with said last week "I want you here with me, I promise I will make it worth your wild". Then this week he says "I don"t wanna get between you and your man". He's done this before. It pisses me off. It's wishy washy. Cause I'm having problems with my baby's dad he says all I want him for is an escape. but that's not true. I'm want him but I'm staying with baby's dad just cause of our daughter.He also said "how's it suppose to make me feel that you sleep next to your baby's dad every night?" What's going on in this guys head?-from a guy's view point. Should I just forget about this affair cus when he pushes me away every other day it hurts.