Am I weird for not engaging in sexual harassment?

In a normal conversation a female co-worker of mine made a joke that she was having a sleepover in one of the dorms at the workplace. Then a male co-worker looked at me and said I could join her and bring some dinner.

Of course I got what he meant there, even when another female co-worker grinned towards us and said I wouldn't get what the male co-worker said. But since things went quickly the last time some male co-worker harassed a female co-worker and got fired, I'm just not joining in to such things. Is that prude, even if the one actually harassed is around?
Am I weird for not engaging in sexual harassment?
Post Opinion