When a guy is well dressed, I assume it's because he respects himself enough to not dress with whatever he saw first in his wardrobe and that's a huge turn on. I don't particularly care about designer brands, because price doesn't determine how good you look but I do have to admit that a guy who knows his name brands is another turn on.
I can tell when guys are made to wear those clothes. It's awkward and it doesn't suit you. Not because of your body type but it's like masking who you really are. But I mean, if you think you can do it and you're comfortable with it, who I am to judge?
There's a lot of evidence that points towards moving out of the skater scene into the preppy scene as the right choice. You're profile lists you as 31, which means you're at the age where dress shirts and khakis actually look good on you. Atlanta is a relatively fashion-forward locale, from what I saw over the past summer I spent there, and it definitely helps to get a notch up on the competition for jobs by dressing the part.
You mentioned Hollister. I would consider that more of the teen-early 20s college student scene. It's got the surfer, and even skater vibe attached with it and even though they can make some solid stuff it's a little under what you're looking for. Ralph Lauren is always a solid choice for any age, same with Sperries. But generally it's not about the brand and more about the fit. You didn't ask about fashion advice so I won't go deep into it, but there's no need to drop $$$ on a name if you can find something that fits you better from somewhere else. If you have any specific questions about putting together a preppy look feel free to drop me a line.
What do other men think about a well dressed man? Real men aren't afraid to complement a man on his dress. They'll take you more seriously because you're good style is an indicator that the rest of your life is solid as well. Women do the same thing. A man who dresses well is a man who pays close attention to himself. He knows his body and treats it well. That says only good things about how he treats his woman.
If you are as old as your profile says, then the women you are dealing with at your age should be old enough to have move past the trend clique deal and respect that everyone has their own individual style. Wear what you want and anyone who doesn't respect your own personal style is a sheep.