It may not be either of those. Shortness could be a factor, but not always. I'm 5'3" and for me 5'7"-5'9"-ish is ideal height for a guy. I can't tell you if unattractiveness is a factor, since I don't know what you look like. But it could be something completely different, like your body language. If you tend to be closed off (crossed arms, not smiling, etc) that can definitely cause you to be overlooked.
Not that it helps, but I know what you mean, I've felt the same way. I've always said I'm approximately 90% invisible and have no idea why.