I would. If my partner doesn't appreciate me then why the hell are they with me? I get that we all get on each other's nerves sometimes and occasionally say things to the people we care about we shouldn't, but to criticize a physical attribute is stupidity. It's not like you can change it, and to be ugly about it is downright meanness.
All depends on wether he is joking or just an asshole. Either way you should tell him it bothers you and if he's just being a dick move on and find someone else it's scientifically proven men are attracted to short women so use your height to your advantage lol