Does he only love me when he's drunk?

I've been seeing a guy for 4 months. It was very unexpected for both of us and started out "just for fun". With out too much detail about why, we shouldn't be seeing each other. We're both betraying people that mean a great deal to us by doing so. I have asked him multiple times if we should stop seeing each other and he's adamant in his answer which is always a firm no. During this time when he drinks he makes comments about falling in love with me. Once he told me "Just tell me you love me", another time I wanted to tell him I loved him, but couldn't get it out and playfully said "I hate you". He responded with "I love you too". One day we were having a serious conversation while he was sober, about being together and I told him I needed him to be honest about what he wanted and if we should stop. Which all he could manage to say was "I don't know, but I don't know how to stop either". So while he was hugging me I told him I loved him. His response... He hugged me tighter, said I'm sorry and kissed me on top of my head. He asked if I wanted him to leave and I told him he could stay. We had a great night together but didn't talk about anything after that. Am I wasting my time with him?

Updates
12 d
I apologize for not providing more context earlier. To clarify, neither of us is involved with anyone else; our connection is exclusive. When I reference feelings of betrayal, I am referring specifically to the weight of family expectations.
Does he only love me when he's drunk?
Post Opinion