My boyfriend is shorter than me and less attractive (as people claim). People judge us all the time. What would you do in this situation?

I love him and he's attractive to me and I don't give a damn about his height, but people around me always say that he's "lower class" and that "he doesn't deserve me" or a beauty like me shouldn't be with someone like him. What would you do if you were in my place?
Updates:
Most Helpful Girls
What Girls & Guys Said
113Related myTakes
Click "Show More" for your mentions
Most Helpful Opinion(mho) Rate.
Learn more
Learn more
Most Helpful Guys