Because saying "let's just be friends" feels like it's nicer than being blunt. I personally think being honest and direct about it is better (which means I won't pretend I want to be friends if I don't), but that's why.
Also, maybe they did want to be friends, but they ended up feeling more awkward about the confession than they thought they would, so they drift away even though the intent was there. Just a possibility.
Because women are typically conditioned to be more passive and mindful of the feelings of others, especially men; I was told countless times growing up that you HAVE to be as gentle as possible when rejecting a guy to spare their ego, be that by giving them the "Let's be friends" bull or by making it all about how I'm actually the issue. It's social conditioning really.
Personally I'm upfront because my dad ended up teaching me different later on in life.