How would you respond after being accused of staring at church?

Yesterday I got met at the church by a sheriff's deputy and the pastor (wasn't in any trouble), one of the worshippers complained that I was staring at them and their family. I don't even remember why or even staring at anyone since I really don't like eye contact (I'm a neurodivergent person). It made me feel uncomfortable and embarrassed, so after Sunday school I left and went home. My brother wants to call the church and cuss them out, which I really wanna let it go because I don't see it as a big deal for that kinda fuss. However I don't think l will return. How would you respond if you were in my situation?
Updates
10 d
It's a Baptist Church if that helps the situation
How would you respond after being accused of staring at church?
Post Opinion