Probably because you're attractive, but they don't want to look like they're staring.
They look at you, then when you look, they break the glance so as to not seem creepy by staring or something, then when you look (or walk) away, they look back again because they still want to check you out.
I am very guilty of this.
We're shy :)
They want to see how you look from the frontal view, as well as the back view.
He looked away because he didn't want you to see him looking...guys that are shy usually do this.
It's not a secret, its a simple explanation.
First of all, we're obviously looking at you because you are attractive, however, societal norms tell us that its not right/creepy/perverted to just sit and stare at you so, eventually we look away, sort of like when someone catches you looking at them. It's just awkward. Then when you walk away, we try to get one last peak before you our out of sight.
What would you like us to do instead ?