I like him. But I seem to be in a minority, at least amongst my friends.
I think the thing I have for him is because of his work rather than him.
The first film I saw him in, and most probably the reason I like him so much was 'Enduring Love' and I really liked the integrity he had as an actor. Although now - he's seen as more high profiled, I like that in many interviews and such he gives credit to his earlier work - and is still selective about the work he chooses.
I think this, to me, makes him seem more attractive than he necessarily is.