People do this in general. It's a natural response to being seen from a distance by the person you're looking at. Kind of like a Lion stalking it's prey, the Lion stalks... let's a say a bunny. Bunny doesn't notice anything. But once it does, the behavior of the Lion immediately changes, from relaxed to urgent.
When you're looking at someone and they catch you looking, and you keep eye contact, they assume you're staring at them, and they think it's rude or they ask why you're staring. Sometimes its awkward. So people just avoid it.
Think about peeping Toms, as soon as their spotted, they run away. So when you're looking at anyone, you're in your own private little moment while you look at this person, when caught you just look away, natural reflex.
fun fact: Japanese people don't look each other in the eye at all, it's actually a sign of disrespect.