Long distance boyfriend joked about what he did in bed with his "Other girlfriend" but apologised saying it was a bad joke, Should I worry?

So my long distance boyfriend and I have been together for 10 months and I was visiting him over the weekend as we both had free time and tomorrow is my birthday. We did some new things with each other in bed and was discussing what we enjoyed afterwards as we were chilling out. I told him that I had enjoyed a particular thing he had done and mentioned it was the first time he'd done it and would like for him to do it again. He questioned it, saying he thought we'd done it before and when I said we hadn't he just said "Oh must have been with my other girlfriend" I must have looked horrified as he quickly laughed and said he was sorry and that it was a bad joke on his part and that he loved me.

I was kind of upset but didn't say much as I was a little shocked as he had never joked about something like that before. I know I'm not attractive or in shape, and I had been cheated on before so I'm pretty sensitive about that and we both have the same view on hating cheating and being honest with each other about not leading the other on if we were no longer feeling it. We both have quite a high sex drive and I know he really struggles with it and not being able to see each other regularly but we make up for it in other ways and are in contact most of the day.

I ended up asking him about the joke again a couple of hours later when we were in bed about to sleep as I think he knew something was up as he started being a little clingy and asking me if I was feeling okay. I ended up crying and he held me and apologised for the joke again and telling me he loved me and saying he was sorry for how he made me feel untill I did stop crying.

The problem is that part of me still feels like it really could be a possibility that he has another girlfriend and I can't shake it.

Should I worry or is it just me being paranoid from my previous experiences?

Updates
1 y
So he also forgot that it's my birthday, now I'm not someone who likes to make a fuss about my birthday but it kinda sucks that he didn't even acknowledge it untill he texted me about my day and I mentioned my father wanting to take me out today. He asked why, when I told him he just replied with "Oh S**t sorry... Happy Birthday"

It makes me feel like he really doesn't care as much as I thought he did, and I feel pretty upset
Long distance boyfriend joked about what he did in bed with his "Other girlfriend" but apologised saying it was a bad joke, Should I worry?
Post Opinion