There is only one person in the elevator, you've never seen them before, and it's the only elevator. By pushing the button but then not getting in when it arrives, you are telling them you would rather wait for them to get off and for the elevator to come back, or take the stairs, rather than ride a few floors next to them.
Besides something obviously concerning like a ski mask or a shirt that says "kill all men", I don't think there's anything someone could wear or any way they could look that would make me not get in after I've already pushed the button and they know I intended to get in.
