Of course not silly. What does this mean?

So we live in different time zone. Whenever we text each other either he's at work or in the gym. One time I asked him if my bothering him his replied was of course not silly. What does that mean. He thinks I'm silly or he thinks I'm asking a silly question?
Of course not silly. What does this mean?
Post Opinion