I think it can sometimes be a turn on depending on the girl. If you have the looks and the personality to pull it off it can look good, but don't go overboard with it. I really don't like it when people (guys or girls) wear fedoras every single day. It just seems like you're trying too hard.
I know the type of guy I am and I tend to like a certain type of girl. Seeing you in a fedora would tell me you're probably not my type. I would think "hipster". You might look cute but otherwise I wouldn't give you a second thought. No offense.