I agree. It's really his problem if he gets his hopes up even after you turned him down. Probably best though in that case to respectfully let him know you need space, just so that it doesn't look like you're randomly shunning him
Scroll Down to Read Other Opinions
Even if they haven't asked you out again or done anything wrong? If so, why isn't it rude?