Most Helpful Guy
I think having privacy is very important, his trust issue is very common among guys and girls. Have a sit down with him not an argument but a serious calm conversation, if he tries to apologize tell him thats not what you're looking for tell him you're looking for him to not be so distrusting of you and be more secure about the relationship, don't detail the times that It happen just tell him you know its happened. Try and make it as little confrontational as possible but as serious and firm as you can make it feel like he's being broken up with. Let him know that if this keeps happening your relationship will be damaged beyond repair and a break up is the only solution.
Most Helpful Girl
You need to confront him about this, because he's obviously invading your privacy and does not respect you at all, considering that he's carrying on with this bullshit in secret.
What else is he going to go through in the future? Go through your bags? Scan your PC/laptop? Sniff your underwear?
Guy is extremely insecure and his behaviour is not acceptable. Try putting a password on your phone and see what happens. Chances are he'll get pissed off at YOU and start accusing you of things, because he can no longer check who you're talking to.