Most Helpful Girl
Just keep wearing what you like to wear. Why change yourself because of some guy. I used to do that all the time, but I got tired of playing 'keep up'. It really is tiring, think about it, you are constantly wondering if he will leave you or not like you as much if you don't go with what he feels you should do. I don't even think the comment was knocking your current wardrobe. I think he was trying to compliment you or just make an observation. Most girls I know wear sweatpants, sweaters and pj's all day long. Maybe it was refreshing for him to see someone who dresses up.
I would just keep wearing what you want. If you want to try dressing down a bit, that's okay too. But do it because you want to. I know that dressing nice everyday does take a long time. So it can't hurt to take a day off, but that doesn't mean you have to be uber casual in jammies. You could just integrate some more casual pieces into your wardrobe :)