Is being invited to dinner at his house for the second date, a red flag?

He acted respectful and courteous on the first date. However, I'm concerned that he's asking me to go to his house on the second date. He did give me an option to go some place else at another time if I didn't want to go to his house for dinner and I said that would be best.

Should I be concerned about this? Does this mean he just wants to hit it and leave?
Is being invited to dinner at his house for the second date, a red flag?
Post Opinion