The reason why I downvote people is because I disagree with them. Often it is a strong disagree. If it's a minor disagreement, I tend to just move on and not care.
Why don't I comment something to explain why I downvoted and/or disagreed with them?
It can be a big pain the butt when you have a person get all worked up and bring anything you say as a personal attack on them. Some people get way too triggered from disagreement and aren't looking for a discussion. Those are the kind of people who will rage PM you to say "why'd you downvote me?" It's not fun to deal with those people... Even more so if they're the kind to hold silly grudges.
Likewise, writing a comment and getting in a debate can be tiring. If I left a comment on the thread already, there's no point in me arguing my own stance when I already did it somewhere else. I'll downvote the person I disagreed with, write my own opinion, and save the trouble.
I've been downvoted excessively before too, but I really figure there's no point to feeling bad about it. I mean, it's just a number, and people downvote for the silliest of reasons. Like I suspect people going on this thread won't read this far and will instead downvote my post anyway haha.
Who knows, maybe you have some haters that love to downvote you? In any case, I wouldn't worry about it.