Who gets to decide if something is offensive?

I often read comments where someone will tell someone else who is offended by something, "that isn't offensive" or "you're overreacting." How does someone else get to decide what offends someone else exactly? Whatever religion, race, gender, culture, nationality, or country we are apart of, certain things are going to be offensive to you as often said by others who don't either know what is considered offensive to you, your culture, race, etc or who are unaffected by it and therefore feel by that reasoning, it shouldn't be offensive. If someone flat out tells you "this is offensive to me," why do you think they are lying to you or that worse, you should keep doing the thing that offends them whether you agree with it or not.
Who gets to decide if something is offensive?
Add Opinion