He rejected me because I'm not fit?

Anonymous
So I went on few dates with a guy who is extremely fit. He trains, works out, eats healthy, he has a beautiful face and cares about his appearance A LOT.
I, on the other hand, am not AS fit. I'm not chubby either nor skinny, I'd say my body is average. I have a back problem (got it years ago from an injury while I was training a professional teakwondo) and I practically can't do any sport anymore. I also can't do hard fitness, because I would risk getting injured again. I only go to gym sometimes. But I eat healthy and all that stuff, it's not like I'm not taking care of my body at all. I have a pretty face (at least I get told all the time) and in all honestly I almost never had difficulty getting a guy.
Anyways, he hasn't made any move on me and I started to wonder why. In the end he told me that he is not over his ex yet and that he thinks I'm the coolest person ever and that he would like us to just be friends for now, because he doesn't want to fuck things up. I kinda bought this bullshit and agreed to stay friends. But then he told one of our mutual friend (he didn't know that that friend knows me) that "I would be perfect if only I was fit" and that that turned him off, that he wants a fit girl.
How is this supposed to make me feel? I honestly feel fat and ugly now. And I think he is a jerk with an unrealistic standards. Is not being fit seriously a dealbreaker? I considered starting with hard workout even though it will reawaken my back pain and just screw everything just so I would get fit for him, but then again, is it worth it?
Updates
+1 y
I agree I overreacted calling him a jerk, you are right, just because he has some standards that I don't meet doesn't make him a jerk. But he was being a jerk for leading me on, though. Great comments by the way, thank you guys!
He rejected me because I'm not fit?
42 Opinion