When men wear women's garments why we ridicule them? Probably because he is associating himself with femininity and it seems that being like a woman is being less than. A gal dressing in men's garments is seemingly acceptable because it's associated to masculinity and it seems being like a man is being more than. Basically it appears that a gal is taking a step up being like a guy while a guy is taking a step down being like a gal.
you like wearing ladies clothes and dont like the fact some people in society think its weird. is that it? well wear what you want but men are expected to behave as such and act accordingly which does not include wearing a flowery dress. for the same reason I wouldn't by a tampon and ask the shop assistant how I should insert it, or wear make up and paint my nails.