How to tell someone something bothers you?

How do you tell someone that something is bothering you? The person asked me if what they did bothered me and it didn't at the moment, but the more I think about it the more it bothers me. I'm scared to tell them that it bothers me now, because I don't want them to be upset with me. I'm also scared to tell them, because I don't want them to lose interest in me since I can't accept what they like to do.

Can someone help me with this please?