I think it's sexy if the guy wears it with the right jeans, and has the right look for it. By look, I mean choice of style - a guy who is more professional dressed will look hot in it =] But I've seen hard rockers and emo styled guys who do it - and I think it looks awful.
So if you are a professional-type dresser in the first place, it should look pretty good.
Doesn't matter what they say, if you like it, wear it like a champ.
That being said, I like to wear dress shoes and jeans, but only really when I'm wearing a button shirt or sweater or something up top. T-shirt generally means no dress shoes (for me). But I think it's a great way to get that classy/casual look.