If you are uncomfortable with it, talk to him about it. Seeming clingy isn't nearly as important as your comfort and communication in a relationship.
I'm a stripper and I can tell you sweety that different men come for different things. Some men come because they feel like they should be able to enjoy themselves and watch women dancing around naked. Some men are not happy at home, or may not be fulfilled sexually at home. Some men are lonely. And so on. I would be concerned as to how much money they have to spend. I work in the VIP lounge, and I can tell you that if you are a concerned girlfriend you may want to keep your man away from that area. More explicit things happen than simply watching girls dance on stage.
I don't know if you're comfortable with it, but maybe you want to talk to him about coming along? It depends on how open you are, but maybe it can help you become more comfortable with the setting. I have had a few men bring their girls with them, and I always try to make them feel comfortable, I flirt with the girls and make them feel welcome, I get them involved. It's more or less all about the women when they come to the club, because I understand the insecurity they may be feeling and I want them to be part of the picture of "sexy and desirable" so that it helps reignite that attraction in the guy's eyes if they are going through a dry spell. I got one guy's wife to come on stage with me and dance. She was red in the face, blushing, but she had fun and her husband was turned on. I would bet good money that they had great sex that night. Sometimes it may even mean the spark lights up again and they come to the strip clubs less. Other times not so much.
A lot of the strippers where I work develop clients and actually have conversations with these guys. I have men who come to see me specifically, who like talking to me and sharing things about their lives, some of them even build an emotional attachment in a way. So I'm not going to pretend that its always just them looking then leaving. That would be lying to you. More importantly though, things can get pretty explicit, and I know most of these guys aren't telling their ladies what's going on here. So you should inquire and say you would like to know what's going on if he goes to the private lounges or not, and just talk to him about it.
The fact that he didn't tell you and you were just cluelessly wondering why you were never invited is a bad sign to me. Not to say he'll leave you soon, because a lot of men like to have their cake and eat it too, but it's not honest or fair to you. What we do is not a vanilla business, it's not something that should just be assumed okay in a relationship. He should tell you that he's going out and staring at half naked or fully naked women, maybe getting lap dances and touching them or more in the VIP. He owes that to you if you both are on the same page of commitment. Talk to him about it.