Is it illegal to be outside someone’s house?

So I was just waiting for one my friends and it was on someone’s street and there was a big fence between me and this persons house so I was next to the fence waiting. Then some guy comes out from one of the houses next to it like 3 doors down and comes up to me and says “ oi what you doing here this is my street” and I said I’m just waiting for my mate and he said it really aggressively and he was like “ well hurry up” and I’m thinking it’s not a crime to be on someone’s street but I didn’t say anything. It really shook me because I wasn’t doing anything wrong I didn’t even know this guy. But is it illegal or something to be waiting on someone’s street like that? Because so many kids are always on my street who don’t live there and nobody says anything. So what was that guys problem?
Is it illegal to be outside someone’s house?
Post Opinion