Sweet looking to me is polished, conservative, innocent, etc. It's not a bad thing, but you'll never please everyone. It's like being blonde - most people will either like it or be neutral about your hair color, but there will always be someone who tells you to dye it brunette. In the same way, not every girl is going to go crazy over a "sweet-looking" guy, but do you really want attention from girls who don't like it? They'll just pressure you to change.