I do this on a regular basis. I'm very unconfrontational, so when someone does or says things that bother me, I try not to make a fuss because I'm afraid of what their reaction might be.
What ends up happening is that I think about things in the past that I wish I would have done differently. It bothers me, and makes me angry- moreso at myself. It takes a huge blow to the self esteem, which is not good because you have to lack some in the first place in order to be so afraid to express your feelings. Eventually you sorta just become this ticking time bomb of suppressed emotions.
Basically what I'm saying is to try not to let it rule you. Don't let it become a pattern. You really need to go back to this person and tell them how you feel, because if they have any ounce of respect for you, they will apologize for it, and you will feel so much better. If they can't simply do that, then you will be risking getting a bad reaction, but don't be afraid if it happens. You're only human too, you have the right to set boundaries for others to respect, else they'll just walk all over you because you let them.
Take a stand for yourself, try to tell the person in a nice way that what they did bothered you, and you'll feel lots better.