Search Results: her boyfriend around my child

So... my boyfriend and I got into a huge fight, primarily over him not compromising on where and what we were going to be doing for the weekend. Long story short, the argument could have been resolvable but then he lashed out and made a comment about my kids being bad... in anger I hung up the phone. 2 hours later, he texts me an apology, but I haven't responded to him since. I felt his comment was immature and irrelevent to what was going on between two adults. It made me feel as though the new man in my life just summed up everything he actually felt about my children. At this point, I'm still pissed and feeling that this whole thing between me and him was a huge mistake, especially me bringing him around my children. This would have not been the first time me, him and my children have done something as a family. I was truly offended and I felt he was super immature about the whole ordeal. I've considered breaking-up because my children and I come as a package deal. I understand people become frustrated and angry at times and say things they may not mean. But, I felt I had to draw the line on that comment. We still haven't spoken and at work we haven't spoken. I'm not giving the silent treatment, but I need him to understand certain things are a no-go for me, my children being one of them. I'm still very angry that he came out his face the way he did. I would have more appreciated and would have been more forgiving to a a phone call apology. What now? I'm just wondering what is going through his mind. Nobody is saying anything...

There are 2 in particular that have remained family friends. I do not expect my brothers to stop associating with someone just because we didn't work out. Besides there was no "serious" feelings envolved either time. We just weren't a match.

My brother-in-law is a local musician and these guys date other girls now and take them to his shows and even come to Christmas and Thanksgiving at our house. I don't have a lot of dealings with them, but I am not rude when I see them, and I do not go out of my way to avoid them. I'm just indifferent.
I've been dating my new boyfriend for a couple of years and he is pretty jealous anyway. But just recently he learned that I have went on dates with these guys before now my life is extraordinarily boring and lonely. We are a pretty close family. Now he doesn't want to go to holidays or family gatherings with us anymore and gets furious if I do. I can no longer go to my brother-in-laws shows. But the worst thing is how he accuses me of keeping them around on purpose, so i will have someone on the backburner, incase I get mad at him. That hurts a lot because it shows me that he really doesn't know who I am after all this time. Plus I am sad that HE feels like I would do that.
He has an exwife and child that I may always have to deal with, but I accept that... even though he talks to her regularly AND I even found out Later that had slept with her a couple times when we were first together. If I DO say anything about it, he tells me stop bringing up the past. I don't think It is fair that I'm willing to deal with that but he thinks I should have to cut off my family or give them an ultimatum to cut their friends off or I can't associate with them anymore.. even on holidays. I love him but I am getting so sad. I feel stuck between a rock and a hard place. What should I do?