There are a few different issues here.
First, the fact that he doesn't remember your phone number is irrelevant. With most people using cell phones, few people can remember a specific number off the top of their heard. You're not 302-555-1234, you're speed dial 4 or "Mary" in the phone book. Expecting him to memorize your specific number, and spit it back out like it a was a question on a test, is unrealistic and insulting.
Second, he doesn't have to be consistent when he tells you what he feels. His feelings are *his* and you have *no* right to demand that he share them. As a general rule, people avoid sharing their feelings when they fear that their feelings aren't respected. "When she [Patricia Sun, a couples therapist] would sit with a couple and help the man to draw out and express his feelings, she found that the woman very often did not really want to know what the man felt. The woman just wanted some words about feelings that she could manipulate so as to create a sense of control over the situation." link