It's natural and therefore normal to have the thin fine, almost invisible hairs. By far most women have it, and also on other parts like chin, jawline and lower lip. Usually it's very fine and practically invisible, like yous say only noticeable if you're standing close. I also noticed some women cake up their face so much, the hairs just disappear, but a caked up face is just ugly.
You could do waxing, or maybe laser treatment, but I'm not really bothered by it. You can read the following link for more ideas and views on this: link
Its not THAT big of a deal if you can't see it but I personally find it unattractive. If its really visible its a HUGE turn off in my opinion. It stems from the fact that facial hair is a man like attribute. Straight men aren't turned on by overly masculine qualities. From a biological perspective we look for the most feminine girls because it shows fertility and health. Traits such as large breasts and small waits are found to be attractive because those physical qualities are by-products of estrogen. Same thing for women, they look for broad jaw lines and physical fitness because those are by-products of testosterone.
As a girl, I feel "ANY KIND OF HAIR ANYWHERE THAT DOESN'T BELONG", should be taken care of. Girls should know better than to even leave a FRAGMENT of a hair on their upper lip...Very unattractive and a real turn off to any man. Of course a man should be "bothered". When you are out there in the dating world, everything counts..even the hair on your face. After a girl is married, then that is a different story.
I made the decision to stop removing it due it being time consuming, and frankly a pain in the arse in general to remove due to my very sensitive skin. I have brown hair, so you can see it if you are close enough, and I wear very light makeup so it's never really covered up. My boyfriend noticed it esterday and pointed out I had a moustache. I asked if it bothered him, because I can barely see it or feel it myself, and he just now sees it six months after I decided to stop removing it. He said it didn't- he just thought it was cute, but was trying to be funny. We actually had a good laugh over it. If it's only a little bit of peach fuzz, I don't see why they would be extremely bothered by it. In fact, in the novel "Madame Bovary", one of her lovers actually considers "the slight dark down upon her lip" to be endearing. Don't see why that can't be true and a relevant opinion in the modern era.