One is a dis and the other a compliment in my book. That outfit looks good on you would be me commending their clothing choice. You look good in that outfit would be saying they don't usually look good, but the clothes they have on are making them look good. I would choose to compliment them of course and say that outfit looks good on you which would imply that they are already hot, but the clothes bring out the best in them :)
I use the first one if I'm trying to hint that I find someone attractive, the second is one I'd use if I wanted to keep a distance from a girl that I wasn't really into, but didn't want to say something like, you look just awful.