It's only too late if you don't feel the same anymore. If you're not in love with him, and you want to find Mr. Right, you're going to have to dump him. Shame he's probably going to hate you a lot and not be able to trust women from now on. Well, eventually, he might. Anyway, it takes some guys longer to tell a woman he loves her because he's use to just a physical connection with a girl and not emotional. Other guys have a fear of opening up and getting their heart broken. It depends on the guy, but guys are generally more reserved about how they feel.
People know nothing about love. Sometimes people confuse love with passion, sometimes they think that they don't love a person but after a little while without him/her they realise that they do, but its too late
So think really carefully before break up with him. Remember all the good time you have spend with him. 3 years its a really long time and if you didn't love him you wouldn't stay with him for so long
I have had a relationship with one girl for 1.5 years and I have never said "I love you" when I loved her more then any one. Because I was not sure that she loved me .