Why doesn't he talk to me in public?

I am a senior in college and there's a guy I've been semi-seeing on and off for about four years. We've never technically been in an official relationship, but we still hang out from time to time. The problem is whenever we're out in public be it a bar or a party he barely says a word to me and sometime he'll avoid speaking to me all together. It's crazy because we have we have tons of mutual friends and none of those guys have a problem talking to me, I'm fairly attractive and I don't have a reputation because he is only the second guy I've ever done anything with. Usually after we've both left whatever public outing it we were at and went our separate ways he'll calls me and asks me to hang out with him alone. And when I do he barely talks and when I talk it seems like he's not listening. I don't understand him or why it is he doesn't really talk to me in public, is he somehow ashamed of me (why he would be I don't know), or is that he's just a private person?