No, some guys wear pink shirts and they aren't gay. Seriously, whatever colour you wear doesn't matter.
It's the way a guy carries himself that would denote him as gay-like.
I think as long as you act like a straight guy, there shouldn't be any issues or disputes over your sexual orientation, generally speaking, (I hate to stereotype) but you know what I mean.
I used to work with this gay guy at my other work place and he would check out guys' asses when they would walk by our place (I worked in customer service then) and it was so funny. Anyway, you get the idea. No, I wouldn't worry about it. However, personally, I don't care. But I guess I do prefer certain colours. ie. white, blue, black, red, brown, gray are kinda masculine colours.