Most Helpful Guy
Even if you ditch him those feelings for him won't go away, if anything they'll intensify. I think if your feeling attracted to him then that's all the reason you need and real friends would understand and support your decision putting aside personal feelings.
Another thing to consider is the friendship you have now, you put that at risk when it goes from friends to being in a relationship. But if you think its worth it go for it.