Most Helpful Guy
I would probably think you were trying to act based on some obnoxious dating playbook you either read or you and your friends came up with:
"Kiss briefly on the lips at the end of the second date."
"Hold hands for most of the third date, give him a convincing embrace at the end, but only a kiss on the cheek to keep him wondering if you're actually attracted to him."
"Another short kiss on the lips to start the fourth date to ease his doubts. But avoid physical contact for the rest of the date to ratchet up the sexual tension, then conclude the date with a long deep kiss."
If you want to kiss, kiss. If you don't want to kiss, don't half-ass it to fulfill some perceived expectation about how relationships are supposed to progress.
That's my feeling anyway. I suspect it has very little in common with the vast majority of people's expectations, however. The norm for dating adults is supposedly to have sex on the third date and no later than date five, he probably was expecting more, and you've thrown off the schedule.
If he's into you, it really won't matter. Be true to yourself and go with what you're comfortable with, and if a guy isn't okay with that, find a better one.