Most Helpful Guy
I mean, sure, it's a big difference. At the end of the day though, a height difference doesn't really matter when it comes to who you love. You can still operate as any other couple would. The only thing is that people might, although I would think it'd be unlikely, give you odd looks occasionally. Then again, that's no different than how some people would view an interracial couple, and there's nothing wrong with that either.
In short, no, it's not too big a height difference. Go for it. :)