My boyfriend of two years still is "unsure" if he loves me, should I give up?

We had a really rough start, where he told me for sure he loved me when I wasn't ready. I told the truth to him, saying I hadn't fallen in love with him because I just wasn't ready since I wasn't completely over my ex. I wanted to be fair to him. We were on and off and then finally we split up and got with different people. he fell in love with this girl he was seeing after 3 months. Eventually, he went overseas with the Air Force and we ended up talking again. I visited him twice in Japan - half as a vacation and half to see him - spending a combined total of 3 months there with him. We talked every day, texted, called with vonage and finally, after 1 1/2 years of long distance ups and downs (including him saying he loved me twice and taking it BACK both times because he "didn't really truly feel it" ) he has returned to the states and we are living in the same area. 6 months of us being nearly inseparable (not romantically just situationally) and he doesn't show me he loves me, he doesn't "try" to repair the damage he's done, he is hardly romantic at all. I see nothing in his eyes that comes close to cluing me in that he feels something more than just good friends. He says he "cares for me" and that he is afraid to tell me he loves me after the last two times. But he just doesn't show me and I definitely can't feel it. Just this week, he visited family and didn't call me once for four days. We talked about it and he apologized, but really didn't improve very much. He used to call me every night just to say goodnight and to tell me he was thinking of me. I'm getting tired of being in love by myself. I'm tired of putting all of the romantic effort into our relationship and getting no return love and passion. When we make love, it's more of playing around and quick get it over with type of sessions instead of the emotionally charged sessions I miss. When do I cut my losses and pick up the pieces? I'm afraid to move on from him because he's my best friend and I am completely in love with him... I'm interested to hear a guy's opinion.