Can someone tell me, what's wrong with this?

So this situation happened, I met this lady in my office long time back. She is around 40-45.
She is a single mother, and she has a daughter around 20 year old. She is almost my age, just a little less.
We used to talk a lot in office and soon we started going out for meals. It turned out, she is also not looking for any serious relationship. We started hooking up and now, we both are having good time. Even her daughter doesn't mind me being with her mom. We both know, it is only for sexual satisfaction for both of us.
But in my office, everyone found out that I am sleeping with her. They started bullying her and making fun of us.
I just wanna know, what's wrong with this?
We both are of proper legal age, and we both fully consent what we are doing. But these people, just ruined our work environment.
We both are looking to change our job.
I feel really bad for her. People are really harsh on her. Even I heard it, someone once said, I take turns on both of them.
I just wanna know, is it really bad that people are going so much mad about it?
Her daughter knows about us, and we both are really good friends.
Can someone tell me, what's wrong with this?
Add Opinion