I don't wear a lot of polos...but I do wear a lot of long-sleeve dress shirts outside of work. I usually unbutton the cuff & roll up the sleeves above my elbow when it's hot, & I wear a solid color tee under it to give a "contrast".
Overall, I don't seem to have any negative reactions by the females. Overall I definitely get "double take'd" because that style of dress makes you stand out in the massive crowd of guys wearing tees.
Plus, polos definitely do flatter your body, so if you're in decent shape women do notice. :)
So...i say women like that style. Sophistication+flatters the body = girls think you're yummy.
Not my style but if it's working for you and you are comfortable then go with it. In general though, it's good to have a little contrast. You look like you are in very dark solid colors. It has no flavor, no character.