I am in agreement also. When I give advice, regardless of who asks the question, I try to be as open minded and unbiased as possible because I want to give fair and impartial advice. In the comments area I will sometimes see people state, "that makes sense" or "I agree 100%", etc, but as I look at it , I will get a barrage of down votes just because I won't support a one sided view.
I think the down voting goes beyond the like or dislike of a particular person, but quite often people will ask self serving questions and of course their followers will posse up and support them by giving them the answers they are seeking. Someone else will come in and give a more reasonable answer and they get down voted because they didn't support the same view as the person asking the question.
Recently I received a best answer vote from the person asking a question but because it went against someone's one sided views, I got 11 down votes. Of course the question was dealing with issues of race, and some people tend to be very biased in regard to such a sensitive topic.
Even if the up vote or down vote feature isn't to be removed, I do believe it should be amended in a way where as it forces the person to state (without anonymity) the reason they disagree, but then again, some people would post a bogus answer if they're intent on down voting, but the feature alone would probably discourage a lot of unfair down voting.