Is my boyfriend in the wrong? should I leave?

So a little backstory, I hooked up with a close family friend a few years ago. My dads best friend is dating his mom and our families are all close family friends. I just recently told my boyfriend about it. He did not take it well. He gets very angry whenever my parents talks about him or about his family.
me and him (family friend) are not in any contact at all and never talk. Today my dad invited him, his mom and brothers over to their house. My boyfriend was over and went back to our house and texted me at work flipping out on me.
He told me not to text him. And he told me that my past is bs and it’s all my fault that he has to be reminded of it all of the time.
in my opinion I think this is completely uncalled for. I think it’s out of line and I don’t know what to do. Me and the family friend never talk and I can’t do anything about him being in our lives because his mom is dating my dads best friend who is practically his brother and an uncle to me. I’m completely lost and he’s making me feel like everything is my fault and I’m a terrible person. Please help.
Is my boyfriend in the wrong? should I leave?
Post Opinion