Now, I have a huge problem with this whole scenario. You are letting yourself be used by a man who has already done this to one other woman. And yet, you and her fight over the man, rather than sympathize with each others' griefs.
The only way he can make this right is to marry one of the two of you, and be a non-sexual friend with the other. But he has to be in the picture of both women for the sake of both children, as of right now.
This is one of the primary reasons that polygamy was outlawed. Although, old-timey laws also stated that behavior like this with non-marrieds, especially when children were at stake and one child would be deprived of a father for the sake of another, was also horrendous. So under old-timey laws, he'd have to marry one of the two of you or else face jail time.
Those laws aren't being enforced anymore.
But as a rule of thumb, here's some advice that may sound appalling in this i-Generation age:
It's never about what you want, but about what truly is best for your children, that should govern your actions as a parent. If that ends up meaning in the long run that you have to let him go, and find another man who is a better role model for your child, consider the options.