Your not ugly brother. A wise man once told me - girls care about looks as much as you care about how smart they are. As long as you are not UGLY AS HELL, as long as you have a good personality, a good job, and aren't crazy, no one cares how good looking you are.
But what does it mean to "be ugly"? Beauty is subjective! It is often said that "beauty is in the eye of the beholder" or "one man's trash is another man's treasure." There's always people who will find you attractive and there's always people who will find you unattractive. Can't please everyone!
Your not alone sometimes I feel the same way and I have been even told my some girls how good looking I am I have been rate everything from a 2 to a 10 seriously being given a 10 is rare so obviously I was flattered beauty is in the eye of the beholder.