Is it considered rude not to order anything?

I recently was out with a friend and we stopped at coffee shop. While there, he offered to get me something to drink, but I politely declined because I honestly didn't want anything. Would this make it seem that I was uninterested in him? Would this bruise his ego?
Is it considered rude not to order anything?
Post Opinion