Can a "best friend with benefits" arrangement work? does he even want that?

sorry in advance I really dont know how to explain things in a easier way

I've been best friends with this guy for a year now, and I hold him very dearly. We actually met because I had a crush on him, but story goes on and we became just friends. He got drunk a while after I was already over the crush and told me he had some feelings for me in the past, but we talked after and decided that that was in the past. In a truth or dare game, we kissed very briefly. After that, we just got closer and closer, emotionally and physically, we started holding each other more and talk about sex and stuff way more. We even kind of started "fight playing", simply because we were sexually frustrated but didn't want to act on it.
After one more talk, I told him I was way too scared of ruining what he had, he insisted on saying there's no way to ruin friendship with sex, but I just laughed and carried on. After that, he told me that it was all in my hands, that when I decided it was time, we would hook up.
A week or so later, I kissed him by impulse. He was kind of shocked, so just later he told me that he wanted to kiss me more when I left. Another week pass, I said that he didn't need to make me want to kiss him, because I already wanted it. And he kissed me.
But everytime things like that happen, I panic and leave. He told me he was confused later the day, but never told me what about, I just know that it was about us. Ok, the question here is: I don't know if we should hook up. I love him so much, and I'm obviously attracted to him and I want to have sex with him, but I don't know...
I have a very low self esteem, so, for me, he's just being nice and saying things he thinks I want to hear and that he doesn't really want to hook up. I don't know if you can really take anything by what I say, but...
Can a "best friend with benefits" arrangement work? does he even want that?
Add Opinion