If it were me, and I say that because I have in no way intention on inferring "the right thing" to do in this situation, as my opinion is a good as anybody's and therefore immaterial to the the situation at hand; here goes it.
Obviously what comes to mind is that he's talking to another person, a girl. Much to your apprehension, unfortunately.
Fortunately, though, there are scenarios I can share which do no involve him faltering in his loyalty. There is a possibility that there's something going on which he might not want you involved because of infinite possible reasons. e.i. it's a surprise to you, it's something he doesn't want you to worry about, a female friend (which you might or might not know of) has developed feelings for him and he's trying to silently save such friendship or trying to avoid her altogether before you find out and jump to heated conclusions, something bothering him and he doesn't want to spoil your stay together with him complaining about it, so on and so forth.
Calmer, now ?
On another note, if you plan on asking him, make sure he knows you trust him, tell him so, I'd advise you on that because your curiosity could be misunderstood for mistrust on his part, it is indeed a fine line. Gentleness is rewarded, so is the lack of it; you reap what you sow.