Should I say yes?

So when I was 17 I glanced over at this guy (20m) from across the room and ever since, he became interested in me. I reciprocated, but since he never approached me or replied when I tried talking, I dropped it.
I only saw him occasionally but he still continued for 5 years! He'd stand at a distance and stare at me, or at parties he'd hide behind a door and stare ( it's like he wanted to have some sort of 'affair' with me without talking and it was super annoying) . I ignored this hoping he'd get the message but one day he was standing afar and watching me for a couple of minutes straight continuously. I felt uncomfortable, I hid behind simmering for a few seconds to avoid him but he didn't stop. That's when I realised it was getting stalkerish and basically decided to completely avoid this guy after that. It's been over 2 years since I'd seen him.

Over the past few months, I started talking to his younger brother who's actually a very sweet person. He said he wanted to be in a relationship with me and I'm genuinely interested in him as well, but I'm hesitant because of the past with his brother.
Should I say yes?
Post Opinion