How big a deal is height?

The guy I am interested in, and who seems interested as well, is 4 inches shorter than me. he's also incredibly fit (dancer) and I stay active, I'm just not extremely fit. How big of a deal is that?
How big a deal is height?
Post Opinion