Why does he look away when he talks to me?

I interact with a man regularly on a professional basis. When I speak to him he always looks me in the eye as if he is really listening. He is always extremely pleasant and polite. If I break eye contact for a moment and am still speaking, I can see that his eyes drop. Not low enough to my chest, I am guessing my lips. When he answers me or talks to me, he either looks down to speak or he looks off to one side and then will return his eyes to me when he is finished speaking. He is not looking around the room, just looking away Now, what I am wondering is, obviously he can look me in the eye, but why does he only do that when I speak?

Thanks for the answers! You both mentioned he may feel intimidated. What would make a woman intimidating to a man, just being attractive? or would it be more of just something he is feeling on a more personal level.