I think the rugged style does add something to it.. Like s3tupMayne rightly stated, his face has something about it that makes it attractive. However, personally, I think he does need facial hair to set it all in. I don't like him (physically - his acting is great ;D) in Charlie and the Chocolate Factory for instance, but I melt for him as Roux in Chocolate and Captain Jack Sparrow ;) Same goes for Viggo Mortensen. In Hidalgo and The Lord of the Rings I love how he looks, all rough and masculine :P but once his beard goes away the whole face changes. But then again, I think I'm a bit of a facial hair - type girl ;D
It's not his style, it's his facial features that make him attractive. He's got a ridiculously desirable bone structure. Three things he's got going for him:
- Sharp, heavy, masculine jaw
- Very defined cheekbones
- Deep set eyes
All these things are considered attractive on guys (Google image it, you'll understand) because they're all indicators of maleness. Different hormones make different bones grow differently. Testosterone makes the bones in male faces grow denser and larger, so having a defined and deep-set bone structure indicates a good healthy dose of male sex hormones. It's sort of like how estrogen makes girls put on fat in the booty and the hips, which makes them curvy.
So just like guys are wired to look for curvy girls, girls are wired to look for guys with sexy faces. Johnny Depp has that sort of face. Looking rugged just happens to be part of his style.