The controversy seems to be not about authority but about adulteration. What the believers of a literal interpretation tend to forget is that life is dynamic.
For instance, they all seem to accept the idea that God had forbidden Adam under the penalty of death from eating apples. However, what they seem to fail to understand it that once the penalty was delivered there was no longer any reason for excluding apples from our menu.
