It does matter. For example, for me, since I am only 18, age definitely matters. I would not be with someone who is 27 or 30, etc. Just not right. But then if I were 23, then 30 doesn't sound bad or 32 doesn't sound bad.
But as for younger, assuming at 25, I MIGHT be okay with someone being 2 years younger than me, not sure though.