Would it be wrong to quit seeing someone because they have a little kid?

Long story short, been casually seeing a guy for a couple years. He asked me to do dinner and a movie earlier. Now he’s talking we may have to just do brunch tomorrow because his kids mom is still an hour away in slow traffic picking their kid up from him. He said he can order us Texas roadhouse to have for tonight and just chill at his house. He’s only a year older than me and has a three-year-old. All my kids are grown. I mean for god sake’s I have one grandson that’s four. I really like everything about this guy, except for the fact that he has a toddler that seems to be an excuse every time he has to change his mind about something. I know him and the mom’s not together because I go to his house a lot and there’s no signs of no woman ever being there.

Would it be wrong to quit seeing someone because they have a little kid?
Post Opinion