Why is my guy being distant?

I have been seeing this guy off and on for quite sometime and have been pretty much okay with him popping in and out of my life, until I realized he's someone I could see myself with in the long run. during one of the times that he was out of my life, I got back together with an ex. I knew this really upset him and every once and a while he made an effort to let me know that. I couldn't get him out of my head so I knew the relationship I was in had to end. Now I've been trying to take baby steps with him but every time I try to pull closer he feels more distant. I don't understand after he was the one making an effort to tell me how he feels. I'm a college student and he is back home so I know that plays a part but now I'm home from school and he he hasn't made any effort to see me. we have plans this Saturday to go to NYC, should I bring up him being so distant and how should I approach this topic without seeming clingy or desperate?