It's not selfish but rather stupid as obviously, good looking people won't consider average/ugly people since they can "do better" so that person is severely limiting themselves. But it's definitely always better to stick with what you are actually attracted to rather than to lie to yourself and your partner.
It's not selfish, because everyone has the right to choose who they want to date or flirt with. After all it's you who'll be dating them. You can't just date someone based on someone else's preferences , or just because a person judges you negatively for your choice.
As long as you're not disrespectful or mean ( like you stated) then there's nothing wrong with being selective with who you date...
It would be cruel to flirt with a girl and lead her on , if you weren't interested in her at all. That would be mean
Most couples are with someone equally attractive as themselves, but there is the odd occassion where an imbalance happens. This is is usually possible when the less attractive person is extremely secure about their looks and truly believes they are worthy of the more attractive person. Or sometimes it is the attractive person is insecure and thinks they can't do any better