Most Helpful Girl
I think it's a little odd that your brother thinks your ex-wife is fair game. Was your brother ever interested in your wife when you two were together? Was your wife ever interested in him? In my family, siblings don't date other sibling's exes. It's kind of an unspoken rule, but it's a sane one.
It's pretty unusual and I can understand why it'd make you mad. It could even be considered inappropriate in the eyes of whomever decided custody. Maybe you should call the lawyer who handled your divorce and see if an 'unsuitable environment' could be a reason to ask for more custody.