Don't you dare, doubt yourself. Many decent girls, wouldn't care whether men are black, white, olive skinned, yellow orange you name it. It all boils down on the personality. Looks to me, mean very little and I am sure I am not the only one who thinks like this. I have dated coloured guys before and at the current moment I am dating a guy who is light brown. I see many girls holding hands and kissing coloured guys (dating). As for scary, I admit sometimes yeah I get scared but that is because my area is a sh*t whole and is really bad with gangs and all, but in other areas, not scary at all.
Have confidence OK, I don't want you looking in the mirror bringing yourself down because you are coloured. There is nothing wrong with that.