For me this is an incredible deal breaker but people these days tend to do this so much! They are too used to be holding their phone, just checking the phone for no reason at all. To me that is annoying and sends me a message that I shouldn't be there.
So when I find myself in this kind of situation, is there anything I could say to the person? Anything funny that would break the ice and also make the person aware that they are ruining the date by doing that?
Well if the person just checks the time and then puts it back then I think there's nothing bad about it. But if checking the phone becomes a habit then perhaps you could try exhaling a sigh very audibly.