It's really only something I think about if a girl is wearing way too much. If it looks like a parrot exploded on her face, that's not exactly attractive to me. If it looks reasonably natural then the thought of make up never even crosses my mind.
As far as guys wearing make-up, I don't really see a problem with it - especially a lot of goth or emo guys wear eyeliner (and Johnny Depp looked HAWT with it in Pirates!). Typically, I don't usually go for a guy that wears eyeliner, but I wouldn't write a guy off because he did.
Girl, I think bottom line, you need to do what you want to do with your make-up, it's your face girl! BUT, also think of this: Him asking you to remove your make-up is a huge compliment. He's saying he prefers the way you look all natural...meaning he likes seeing YOU. That's actually very super sweet :).