I am actually black myself from England, and I've had a similar thing. I have not really been attracted to black men (I know it sounds weird) so I tried to figure out what it was that caused it.
I realised that one of the reasons was arrogance/ attitude.
Obviously not all black men are arrogant, but I have two older sisters, and when I was younger, I remember them mentioning having a lot of attention from persistent, black men - men/ boys that they weren't interested in. They would call out to them and such. I thought my sisters were exaggerating, until I suddenly became a teenager myself.
When I got approached by black guys, I usually got the sense that they were over confident with me. Some women like that sort of thing, but I find that it puts me off. I kinda get the vibe that they were putting on a front didn't have to win my affection with being kind or whatever, whereas I would prefer a guy to be himself. Obviously not all black guys are like this (and plus, I don't know what they're like in the US).
Secondly, I find guys with some hair more attractive.
And that applies to all races in general. I've met very few black guys that don't shave their heads and out of these few, I do find some of them more attractive as a result. I guess I'm one of those people that thinks a bit of hair suits everyone (is that so wrong?)
Having said all this, I would say that I am not racist in terms of dating a black guy, as my reasons apply to any guy from any race, so I guess if you do the same - break down why you're not attracted to them, and see if it is a trait any man could have from any race, or if it is something specific to a black guy, and that will determine whether or not it is racist.
Some people are quick to jump on the racist bandwagon and will label someone as a racist, when the situation actually had nothing to do with race until that was mentioned.
If you find that your reasons are race related, then I would suggest seeing the individual for his merits rather than the colour of their skin, or the fear of the different culture or whatever it might be.
But at the same time, people are allowed to have a preference in terms of what they are attracted to. Eg: I'm used to being the only 'non-white' person, in my class/ work, so most people I associate with tend to be white, and so I am very much attracted to white guys and all races. The same could apply to you, wherein your ideal of what's attractive is linked to your culture's ideals?