Guys, would you be mad if your partner was having an innocent lunch with her boss?

Guys, would you be mad if your partner was having an innocent lunch with her boss?
Hi, I've been transferred to a new office last week and today I asked him for permission to be off work tomorrow and next as I had family issues to deal with.

He agreed promptly but then asked about my hobbies. I told him I like to go on walks and he asked when I would return to the city. I said Sunday and he's like "oh I thought you would be back by Saturday so we could go on a walk."
I was stunned & said it would not be appropriate as I'm in a serious relationship. He was like "I don't mind. It's just a walk. Or would you prefer we do lunch?"
I said "no need" and he said "what are you afraid of? Your boyfriend?"Before I could respond he dismissed me with "we will have that harmless lunch next week, alright?" Then he rang his secretary and I had to leave.

Guys do you think I am being paranoid or he's just being friendly? I don't know why I attract my bosses. If my boyfriend finds out about this then the work issue will intensify (for those of you who know my former questions and what I'm talking about).

Let me know how best to handle this new problem. Thanks.
Guys, would you be mad if your partner was having an innocent lunch with her boss?
Post Opinion