Most Helpful Guy
Cultural differences are strong. I believe, also in Korea, it's safe to leave your baby alone for short periods of time. Strangers won't ever harm them, and may even look after them and feed them in your absense. You always start your meal after your elders and never leave the table before them.
Tell him you want him to feel comfortable with you, and you'll take as much time as he feels he needs. Ask him if he can meet you halfway, with baby steps. Suggest something small (like just hand holding), or ask him for his suggestion.
Don't rush it. If he's going to be living in America for a while, he should feel learn how to feel comfortable with some of the customs; just give him the space he needs.