I wear beanies in during the winter. My head is shaved; I have to have something on my head to stay warm.
I wear caps during the summer. I have a school cap that I love. I've had it for years, but it's very worn, black smudges, scratched, and the bill has chewed a few times by my dog. I have a bunch of new caps, but I refuse to give up my current cap until it falls apart.
I happen to like wearing Irish caps they provide good sun protection. Go good with just about any casual attire and are very warm and snug and it's a tradition in my family for men. I must have gotten like a dozen or so compliments on the hat. In my pucture when i got it As for hats like fedoras, the fedora should only be worn with a suit period nothing else.
You don't mind the caps, plenty of others don't mind the fedoras. I get compliments all the time on how I dress, fedoras included. I wear them because I like them though, I couldn't care less what anyone else thinks about them.