Most Helpful Girl
... He lived with her and her child...
There is a Big Possibility here, dear, the Ghost and her little Casper are still haunting him from time to time. This skeleton that still lives in his closet, is still open, I feel, and rattling your own chains, I assume.
He may tell you he 'Loves me more than he loved her,' but honestly, this fine line that is getting in the Way of Just You Making you You every day and This a Dealbreaker with him Being... Distant.
Go slow with Joe. It's hard many times when One still has strings attached to not only the past but to another heart from the start.
All you can really do at this point in time, is be there for him, Nurse and Nurture something special you both have going, and always Keep the Lines of convo Open for one another. You need to speak to him, face to face, if he is pulling away so he doesn't go drifting away.
Good luck. xx