Our first date was quick- I don't really feel a connection with him, but I said to myself I should wait may be. Anyways, he was holding my hand from time to time, and we talked about food, culture and stuff like that, with some moments of silence. Bug I noticed he would openly 'look out' from time to time when women pass by, and this disturbed me, though we are not officially dating.
No, fuck no. Very disrespectful! Don't get me wrong, guys will look. It's inevitable, but it's just plain disrespectful if he does it right in front of you. If you aren't the center of his focus in the beginning, do you think you ever will be? No!!
And if there is no connection, just drop him. There's no need in a guy you aren't even feeling to be hurting your confidence by doing that.
Did he go home with you or one of the other women? Men look at women; we are visual creatures. Did he stop listening to you? Did he lose interest in you? You are probably being too sensitive about this. Keep dating and watch how he treats you.