Really depends on the girl and the sweats. If you look like you're just having a lazy day or relaxed day it's cool but if you look like your life's been filled with shit days and you've just given up its not so flattering. I only wear gym clothes or sweat pants on days when I feel happy but want to relax. If I feel bad I make sure I at least put on some jeans.
Don't like it, it looks trashy to be honest. Women can get away with a lot of things but just because that is true, its no excuse to wear your pajamas out in public. If you want to keep it simple, yoga pants will do, or a basic pair of jeans with a simple tank top.
I hold me to the same standards as well, most men have no clue how to dress properly in the first place.