Well like pretty much everyone has already said, past experiences is what does it for me. Just like how a girl will become cautious around guys if she's been played before, guys will do the same thing. Prejudice is a natural thing in people, so nothing is going to change that except for a positive experience.
I think like darkknight said, it's usually the prettier women that make me uneasy. I've seen too many of them act completely stuck up and like bitches for me to be comfortable around one. I know that not all of them are like this, but far too many are. So until I meet someone worth my time, I'll continue to keep my guard up.
I think both men & women are guilty of this. People just assume things about people before they know them, it's human nature I guess. I try not to, but sometimes I guess what someone is like before I acutally talk to them. Sometimes a person's body language, style, etc is a good sign of what type of person they are.