There isn't really a good way to determine this just observing other people's treatment of you and hearing their opinions. I think if you get enough opinions that are similar you can get somewhat of an idea of where you stand, but until then it's really hard to tell.
Personally, hearing sort of the same tone about my looks raised my self esteem. I used to think I was butt ugly, but now I see that I'm at least average to most.
Just don't think that you aren't attractive because you don't get asked out or something like that. Not everybody gets a lot of attention from the other gender and it can be for a bunch of different reasons.
Being attractive to one person, is relative. Every guy has his own preferences... Being attractive in general, comes down to how much attention you attract and how many people of the opposite gender make an effort to get close to you in some way and flirt with you, compliment you, etc.