Keep in mind: bras can be pretty expensive (especially if you wear a less common bra size or have very large breasts), so women tend to own a lot more underwear than they do bras. Many women will wear their bras a few times before washing them, but they'll change their underwear daily. It's not all that practical to have matching sets every day, unless you buy all your bras/underwear in the same colour.
Honestly, it's not that important to me. Bras are expensive, panties are not. I don't own any matching sets. If I'm going out, or there's a possibility that I might be removing my clothes, I try to color coordinate. I have a lot of lacey neutral colored bras that go along with most of my sexy undies. Thats about as matching as I get. Besides, do men REALLY care if the bra & panties match in the moment?
Not very important, I can tell you. I'm doing well to have socks that match.
I surprisingly learned a lot from just reading the responses within this thread. As a male, I agree it's far sexier when they match though I never realized women felt the same way. Definitely keeping this in mind for next time.