Should I break up with him?

Okay, I have been going out with this guy for three weeks, and I know that things will be a bit shakey for a while but I dunno. I mean, At first he was all talking to me and now it's like pulling his teeth to talk to me or be near me. His last relationship was more like an elementary relationship so maybe that is why he is like this, but then again I could be making up excuses. I don't want to be all demanding but he was the one that wanted this relationship and yet he isn't putting any effort into it. I should I break it off or should I confront him about it as best as I can ( cause I might explode and say something wrong)?