It depends on the guy, and how they take care of it.
There are some who have other facial features that the just clash example boy that I dated for years, red head, more of a rusty color, very dark, his beard was 2 toned very light and very dark, it was also extremely coarse, and curly, it was also a little patchy. so coming in contact with it hurt a lot and actually caused my face to get very scratched up to the point that I looked like I had serious carpet burn on my face after a make out session. he also looks like a homeless man who shaves with old rusty razors he found in the trash when it starts growing out.. he can not grow it out.. and it sucks because he loves facial hair, and can't seem to accept that he just can't pull it off well.
But then there are others who look incredibly sexy with facial hair, a guy that I was seeing last summer very well groomed, soft, matched the rest of his hair and it didn't clash with the rest of his face at all, and I know quite a few men like this. couldn't really feel it when we were kissing, it was nice.
But as a general rule I prefer a man with out, just because it looks cleaner, but there are few that I don't care which way they go they look great.