Probably because you're attractive, but they don't want to look like they're staring.
They look at you, then when you look, they break the glance so as to not seem creepy by staring or something, then when you look (or walk) away, they look back again because they still want to check you out.
First of all, we're obviously looking at you because you are attractive, however, societal norms tell us that its not right/creepy/perverted to just sit and stare at you so, eventually we look away, sort of like when someone catches you looking at them. It's just awkward. Then when you walk away, we try to get one last peak before you our out of sight.