Most Helpful Guy
Break it off. Yes, looks do matter, and if you're not attracted then you're just not attracted. It's nobody's fault, and it certainly doesn't make you a bad person.
If it helps, turn the tables a bit...imagine that upon meeting you he found out he was actually a bit turned off and not attracted to you. Would you want him to keep seeing you even though he felt this way?