"No sex this time"? Has it happened before? I would just say "I appreciate the invitation but I have to decline", that's it. You do not have to give him a reason and if he presses for one just make something up or tell him the truth if you want. Don't go though if you aren't 100% comfortable. Or suggest lunch somewhere else (public).