I was my hair with conditioner most of the time, but every now and then you still need to shampoo to clarify. Natural oils are good, but if you don't eventually remove them it'll cause buildup which can be almost as bad as having dry hair with no natural oils.
Shampoo will strip away good oils and other nutrients but that's why you need to space out when you use it. You should only really wash your hair once every other day or every three days. That's also why you need to choose your shampoo wisely and find one that works for your hair.
I'm a lifeguard so I thought I'd try one that worked well against Chlorine but all that did was strip my hair of all moisture and make it dry, coarse and unmanageable. I eventually found one that kept my hair hydrated, full and it's color rich. It took a while though. So you're going to need to shop around, figure out what works for you.
Not all shampoo is bad for your hair, just some, and it all depends on you. Conditioner doesn't clean hair, it's an after affect to give it some body, shine and sleekness. Its not meant to clean, just to offer a little extra hydration and that kind of thing. Trust me, I've tried not using shampoo and your hair just gets increasingly greasy at a slower rate.
Shampoo is really the only thing that will pull all the dirt, grime and grease out of your hair.