That's how my boyfriend dresses and I love it. I don't associate it with being preppy or nerdy because I just enjoy how it looks like he put some sort of effort into his clothes. And to be honest it sends a better message than wearing a tank top, oversized sports jacket, and jeans that are barely on. It looks more mature/classy than an old T-shirt and track pants. I don't think it should change what sort of girls you attract as long as your personality is fine.
So in my opinion stick with it, most boys don't dress like that anymore and I think it's great that you do :)
I think that's nice as it looks nerdy and it shows that you don't care about what others think and that you focus more on building your life than on your appearance like most do these days, I find it all stupid really. It shows me you are intellegent.
I am going to sound like a goddamn idiot but I didn't think guys really dressed preppy anymore. I have ALWAYS loved this style. I love how classy and distinguished the guy looks. It just always makes me turn my head and enjoy such great taste.