Sounds to me like he only wants you when he's horny. Otherwise, he wants you to get lost until he's ready to get some. Not trying to be harsh but it sounds like a FWB relationship just where you have the title of his girlfriend.
It's normal for guys to want time away from their girlfriends where they can hang out with their own friends and just be a guy. It's not normal for him to be happy to have you over all night for sex just to try to make up reasons to get rid of you in the morning. Sounds like he doesn't want you around for more than just the sex. As long as you keep quiet about the arrangement, he'll keep up the behavior. If you don't like it (as you obviously don't), speak up.
Before launching into a conversation about it with him try this:
1: Do not contact him. No calls/texts/showing up at his place etc. See how long it takes for him to come looking for you. If it takes him too long, that will give you an idea of how much he's thinking of you.
2: When you do spend time together, DO NOT have sex.
3: Don't be willing to just spend time at his place. Make him take you out. Not to just spend up his money but make him actually put forth some effort.
4: Start spending time with other people instead of him. Make plans with friends/family and DO NOT cancel them should he decide at the last second he wants to see you.
Essentially: demand in a non-verbal way that he must put forth the effort. If he's not willing to do it, dump him as he's not worth it. If he's just in there for the sex, he'll think you're too difficult and dump you.