There must be something beyond the physical body. If a man loses his penis, is he no longer a man? If a woman loses her breasts is she no longer a woman?
XX versus XY chromosomes? Not that that covers everything either. The words man and woman are very convenient labels for categories that are sometimes fuzzy.
To have a dick and to have a pussy