Is it a red flag if he doesn’t want me around when he’s annoyed?

My newly boyfriend (24M) and I have been friends for 5 years.
We are very similar and very different in ways.
My flirting is more like soft bullying and pulling his leg. He’s extremely sweet and sensitive so sometimes we don’t understand each other.
Takes very little for me to get upset and I only get upset by serious stuff and I’ll talk about issues.
He is sensitive so he takes jokes 100% serious and his moods go from happy to sad real fast and he doesn’t talk about stuff.

We got in a disagreement last night- he also forgot to take his meds so he was off already. He then told me to just go home and not sleep at his house.

Now I’m upset cause I feel we need to talk about stuff and not just separate and leave it.
Like he didn’t want me at his house cause he said things would feel weird cause we were both upset.
Even if we’re upset I still want to be around him
Is it a red flag if he doesn’t want me around when he’s annoyed?
Post Opinion