In all honesty, it's a good thing. You aren't afraid to be yourself and furthermore I am sure you are much more open to other people's styles/personalities which is far less intimidating then a person who strictly follows the latest trends.
Most girls care at least a little about guys fashion because it is their first impression, so it might affect what girls think of you but that doesn't always mean in a bad way. A different sense of style and be creative and sexy as long as it is done correctly!