Ultimately it is both. Physical looks can only carry a relationship so far. If it is going to last I believe that you have to have chemistry on levels. Plus, lets face it, sex with someone that you actually like and you find physically attractive is so much better than with someone that is just good looking. Hope that helps.
First of all, what your friends say on this matter doesn't apply. Base your decisions on what you think, not anyone else. As for looks, the differentiating factor between a friend and a date is some element of physical attraction which there should be some amount of if you are going to take something to the next level. But honestly, looks can fade, a personality is there to stay. If you meet someone that you truly connect with inside, that's absolutely significant.