Is it rude not to go to a neighbors party?

Yesterday a neighbor said he was having a party next weekend and invited me over. I just thanked him for the invite and left it open. I have nothing against him, but don't want to go either, should I just not go? Or should I approach him, thank him for inviting me, but decline?
Is it rude not to go to a neighbors party?
Post Opinion