What do girls prefer a guy wears for clothes on a daily basis? Does it really make "that" much of a difference? I'm sure seeing a guy (or even girl) at any given store in a pair of sweatpants, isn't the most attractive, but are you really going to find him that much less attractive or even worse...turn him down based on that?
Clothes make a HUGE difference. You need to wear clothes that places focus on your best features. Some men look better in t-shirts than polos, or better in button downs, etc. This is the same as pants AND YOUR HAIR! For instance, I know that I look good in dark colored clothes because I have a nice natural tan, dark hair, and dark features. I also have good legs from sports and strong shoulders. So, I normally where nice shorts if I can and have a t-shirt on that shows my upper body or forms around my shoulders. It really makes a difference.
Women look at men's bodies as much as men glare at women. They are just more subtle about it ;)