Should I listen to what other people say?

My response as a guy. I believe you guys can be offered a chance, try not to take other peoples words so seriously. Find out for yourself. Maybe he might be the right guy, and maybe not. Its tough, I know but youve got to see how he really is, get to know him better. SEX SEX SEX. Yes sometimes people do talk a lot, maybe its natural to be worried. Don't stress about it to much thus it can lead to a relationship full of troubles. Be cautious but not to cautious, Just don't doubt yourself I find hope in all relationships. Just because that's in the past doesn't mean you can be his new future. Go for it if you like. BEST WAY to find out if the guy really really likes you. If he's over sexual or if he doesn't respect your opinion on sex. Waiting might be good.
Alright good luck =]

I would say treat it as early warning signs. He might never treat you badly, but if something does happen it will not be a total surprise to you.
Also, don't contront him about the rumours now if he doesn't show abusive behavior towards you.