This is all going to depend on the girl. Personally I don't like it, especially stubble because if you kiss a guy with stubble if can give you beard burn (for lack of better words). I think the majority of girls I've talked to don't like facial hair. However, usually if a girl likes a guy, facial hair will not stop it, they will just think it's cute. Hope this helped!
Personally I love facial hair. But having said that, I also love clean-shaven guys. I love the rugged look, but a smooth jawline is always great too. I think the only I don't really like is if the guy normally shaves, but he hasn't in a couple of days, so it's short and sandpaper-esque. So I'd say if you're gonna grow some face fuzz, grow it at least a week or something, and invest in a trimmer so you can keep it at a good length so that it's not so scratchy.
Five o clock shadow is sexy, anything totally prominent like a beard or moustache, I personally don't go for. It does matter the guy, because some guys just have facial hair...period. Other guys look so much better clean shaven. The color, if you have facial hair, should definitely match your hair color and your eyebrows!