Most Helpful Guy
From the way you describe it, the guy is completely narcisistic about it. He's seeing another girl who has her own flings and casual relationships. Its like he had this friend with benefits but didn't want to stop the benefits until into a committed relationship, only there is no committed relationship. Not the way he sees it. You're just a potential for being one; there was probably no DTR involved was there? You just assumed?
Anyway, he is a narcicist at worst and a shitty communicator at best. Neither is good boyfriend material. And you know this in your own words "I know its bad to stay with him but I love him."
Normally I counsel on the side of following love, even in questionable relationships, since I know people aren't perfect. But I think you got to say enough is enough, ya know? Deep down you know this isn't a tenable way to continue.
It is going to hurt, but you know what you must do.