Hm...if he holds your hand could be attraction or that he likes you
If he puts his arm around you or around your chair..sexual attraction
If he touches your face...attraction and possibly emotion
If he touches your lower back...sexual attraction'
If he touches your shoulders firmly, especially if he also touches or is close to your neck, it's probably also attraction.
If he touches your ass or t*ts...sexual harassment :P
According to wiki:
"if a guy touches your lower back, he wants to grab some of your butt, if he holds you on the middle of your back, he wants to protect you, if he touches you near you shoulders a lot, he wants to be friends with you, but still have benefits, if you touches you stomach, all he wants to do is be with you, if he holds your head when you kiss each other, he doesn't want to stop, if he rubs your arms smoothly, you thinks you are beautiful. if he hugs you softly, he doesn't want anything to happen to you, if he gives you a quick hug, he doesn't care about you often, if you guys kiss for longer then...1-2-3-4-5, he loves you."
depends on the context.