I started doing that. Not totally but just less. I don't really eat junk just whole grains mostly. I noticed I've started putting on weight. My theory is that it could be because I have less energy which is already a problem. Gobbled down some extra whole grain carbs last night and today and I feel hunky dory.
I was still eating things like fruits, vegetables and legumes when I cut down on the grains.