Are baggy clothes a turnoff?

I usually wear a size small shirt and it fits my body perfectly, but recently I tried upping the size to a large and it looks extremely baggy. Is it my imagination or do girls not even give me a single look with baggy clothes on?
Are baggy clothes a turnoff?
Post Opinion