Casual sex with an ex, yes or no?

I'm trying to navigate a traumatic loss, so a serious relationship isn't feasible for me right now at all. I was dating someone when I realized that, and we broke up. He was going through his own stuff as well anyway (dated someone for over 6 years only to find out she was cheating on him for most of it), so I suggested being casual since we seemed to connect really well sexually and figured it was a connection to hit up when I had an itch to scratch.

However, he's been making me feel guilty/uncomfortable lately. We don't go out on dates, but he comes over just to hangout... even when sex isn't an option (like if I'm on my period or have plans). He randomly told me yesterday how much he appreciates having me in his life (even though he's in agreement that we're not compatible). He says he's done casual before, but it's been with friends. I don't want us to be friends. I'm used to being casual with guys I don't hit up unless I want something. So to me, he's just an outlet for fun, and I'm sure I'll walk away from it eventually, which I've actually told him before, but now I'm wondering if he believes me or is hoping for me to change my mind (I won't).

I feel like I'm using him even though I've been clear about my intentions. At first, I figured it was just misplaced emotions. Am I just overthinking things? I don't think he has feelings for me (he says he doesn't and that he's just finds me attractive), but sometimes it feels that way... is it wrong for me to carry on?
Casual sex with an ex, yes or no?
Post Opinion