Is it wrong for me to ask this?

So my boyfriend has this male friend which I'm not really fond of, because

1) He left his girlfriend for another girl but then things didn't work out and he shamelessly asked her to get back together (and she forgave him) (my boyfriend even defended him saying that his friend was "going through a lot")

2) Ever since me and my boyfriend started going out, he was super disrespectful towards our relationship, like interrumping our dates and stuff

One time, the two of them went to study abroad and he teased my boyfriend that he must have some foreign woman who "caught his fancy" (and also that friend almost cheated on his girlfriend because of some foreign girl he met at the bar)

My boyfriend got mad at him for this comment

The thing is... I usually spent time at my boyfriend's house... But now that friend, for personal reasons, is living there with him for a certain period (I hope)

Last time, I went to visit my boyfriend (he asked me to come) and then his friend came too and when we were alone, he asked me if "I didn't have better things to do" (which made me super uncomfortable)

So now I'm not sure about going to my boyfriend's house because I know his friend is gonna be there too

I want to ask my boyfriend to come to my house to spent time together but he rarely wants to go to my house, since he would rather prefer me going to his house

What should I do?

Is it wrong for me to ask this?
Post Opinion