As long as it's not an everyday thing, you're fine! Back in my high school days, I've seen girls wear sweatpants and baggy shirts almost everyday and it was not attractive. Every once in awhile is perfectly okay!
They look cozy and casual. It wouldn't change my opinion of anyone if they wore sweatpants now and then. I do it too if I was up all night finishing a project and want to be comfy for a day as a reward.