Ugh. Both genders can wear either genders' clothes. There is no rule--in the West at least--against that. So it is okay for BOTH to do that.
BUT women's clothes weren't truly made to flatter a man's body. That's why when some wear it, they come out looking ridiculous. A woman can wear a man's clothes and probably come out out looking either good or butchy. But if that's what those individuals want to wear, they can do whatever they want.
I just want to make an additional point: it's not cross-dressing if a woman is wearing women's pants. Just because men traditionally wore pants, those pants were made specifically for women. The fact that they're pants does not mean it's crossdressing.
Men stink or put off a strong scent and women get turned on by the smell of a man. So often when a woman wears her mans shirt the shirt has his scent on it. In underwear you are speaking boxer shorts which just makes sense.
A man wearing women clothes does not serve the same purpose other than perversion.