That's correct, NO! Okay, this is a simplification because not everyone will agree on what is attractive...but most, and by most I mean the vast majority (~80% or more) what is attractive. Babies as young as 3 months will stare at faces that are deemed attractive longer than unattractive faces. This implies that we are hardwired to determine what is attractive.
But that's really the question, what is attractive? There's numerous studies done on this but in my view there are 3 key FOUNDATIONAL features that you must have in order to be considered genuinely attractive (and by that I mean people will complement you on your looks when you are not wearing makeup): eye area/shape, cheekbones, and strong jawline.
Without any one of these 3 pillars (they do not need to be perfect, but they need to be at least "good") then you would fall short of what I would consider attractive and "baby-staring".
Let's take a look at the first foundation: eye area. Look at Adriana Lima's eyes...they are the most striking feature of her face by far. I am not talking about color either (we'll get to that in a moment). What I am talking about is the same. She does not need to strain her face to make her eyes vertically short and they are wide and perfectly spaced apart. But ratios and shape alone won't give the eyes a good look...you need the supporting structures around too. Adriana has good, but not great, cheekbones and has very little upper eyelid show. The distance from the top of her eyes to the bottom of her eyebrows is very small. It gives a mystery to her.
Notice she has the 3 pillars: GREAT eye area, good cheekbones, and very good jawline. Notice that these 3 pillars do not need to be great, but at least good in all 3 areas for her to be considered attractive by most of the population.
Let's compare that with someone who has roughly the same eye color, but fails at everything else in the 3 pillars:
I am not saying the girl above is ugly, but she is not what I would say beautiful. She is decent, but her eyes are not as alluring or striking as Lima's. Additionally she does not have very strong cheekbones and her jawline is much softer than Lima's. Her eye area is vertically taller and the sclera is showing much more prominently. This is due to the poor support from her cheekbones that are supposed to combat this. But if I had to rate this girl, I'd give her a 6/10. Above average but she is helped a lot by makeup. Without makeup she would probably rate 2 points lower.
Next let's look at another pillar: the jawline.
The model above is Native American model Brenda Schad...she was relatively popular in the 90s as a model. She has striking, but not overbearing, cheekbones in my opinion. They proportion her face well and combined with her equally striking (though not like Lima's) eye area and jawline (much stronger than Lima's) it's clear why she was a model. To me she's one of the most beautiful women (primarily during her modelling years) I've ever seen. Again, notice the 3 key pillars: eye area (great), cheekbones (near perfect), and jawline (great).
I think she's better looking than Lima despite having dark brown eyes because her 3 pillars all register as great in my book.
So now let's look at the last of the 3 pillars: jawline.
Margot Robbie has one of the best side profiles you will ever see. Her jawline is perfect with a visible ramus and a strong projecting forward face. People with receding chins will rarely have a jawline...even if they lose a lot of body fat.
Of the 3 pillars of an attractive face, getting a strong jawline is probably the easiest to obtain. The eye area is the hardest to obtain because so much of it relies on the underlying structure of bone, muscle, and soft tissue.
So rating Margot's 3 pillars: eye area (good, not great), cheekbones (very good), and jawline (near perfect).
You'll notice that the 3 models above (Adriana Lima, Brenda Schad, and Margot Robbie) all have some "flaw" that too many girls focus on: noses on all 3 are either crooked or not very well defined and in Adriana's case her lips are the last plump of the 3. In fact, the 2nd picture (the one under Lima's) has a girl who has more plump lips and a more straight nose...but very few people would consider her more attractive than Lima.
So that brings us to what I call the secondary characteristics that makes a face attractive: lips, teeth, eye color, nose and skull shape.
Lips do need to be medium sized as thin or non-existent lips makes it look like you're thinking about a complicated equation all the time. It's also good to have a wide palate. Since I've only talked about women, let's look at men. Here is Henry Cavill:
Henry Cavill is a hollywood heart-throb. He's got great features but his smile leaves much to be desired. Of course the smile does not break his attractiveness...he has good eye area, good cheekbones and a very strong jawline.
The 3 pillars that I spoke about earlier, you'll notice that I never once mentioned gender (although I did show pictures of women). Those 3 pillars apply to men too. It's the ratio of those features (along with the secondary features) that separates men and women.
Here is Kirsten Dunst, and take a look at what feature you think she lacks:
She has thin lips. She has the 3 main features however. You'll notice a lot that I didn't talk about symmetry. Unless the symmetry is way off, it's not very important when it comes to looks.
Of course you don't want too prominent of features. Take a look below:
Her most prominent feature are her cheekbones. She is attractive but her cheekbones support her eye area too much, giving her "taller" eyes when they should be more deep set for her face. Her cheekbones are so prominent that you can see their shape from the front.
Here is her profile
Although I am sure she is wearing makeup, she does not need makeup to give her the cheekbone shadow because they are so prominent.
So the question is: "Is beauty in the eye of the beholder?"
Yes, for the vast majority of people and cases. We can talk about personality and what not but to say that "looks don't matter" is to ignore innate biology. We are attracted to health, not just beauty. In fact, they are often linked because attractive people tend to be perceived as healthy (and some science backs this up). We do not want ugly babies because it implies a baby that won't have high survival potential.
So why are so many people together? Well, many people don't want to die alone so you hook up with your "looks match." Having a baby is better than not having one, right? At least that is the general perception. But beauty is not skin deep...I've read too many studies, articles, and spoken to too many people to believe otherwise.
If beauty is skin deep then why doesn't the 5'4" 300 lb. woman in a wheelchair have men clamoring for her? She has a great career and is supremely caring.
So while yes, there are always exceptions we just have to remember that beauty is not as subjective as you might think.