If a guy is saying the sister is ugly compared to the guy then he is out of his mind! Why is he finding the brother more attractive then the sister(if you are the sister, hello! I don't care what the brother looks like). I think the friend is attracted to the (your) brother.
who said that wanted to mess with you, and I guess he did... by the way, is this a serious question ;)
In that context, it´s obvious to me that he said it for you to hear it and start guessing, as you are doing. At least, that's what I would have done if I wanted to mess with one of my friends litlle sister.
So, it means he kinds of like you, or at least that he thinks you are cute. You can call it an indirect way of calling your attention.
It means exactly like the question reads. People are curious how one person in the family got the good genes while the other sibling got the bad genes. People expect everyone in a particular family to all be good-looking or bad-looking. It's complicated, and it's called genetics. The "ugly duckling," sister, probably got all the bad recessive genes.