Most Helpful Guy
In my opinion you need to be firm with him and tell him this is the way it is or we are through. It sounds as if he is just in it to use you and your body for his own gratification. In my opinion you should cut off contact with him completely if possible and find someone who is better for you. Before this talking to a psychiatrist probably wouldn't hurt. After that what matters in the resolve in your heart to move on and find someone worthy of you. If you are able to move one try to find a guys that treats you well, and be good to each other, above all make sure you both have the same expectations for the relationship and communicate well.
Most Helpful Girl
Follow you head not your heart. He is clearly only using you so don't even think of going back to him if can't even wait for you. Plus you said the relationship was toxic already so follow your head don't let your heart win.