Anecdotal (sometimes called “testimonial”) evidence is unreliable for a multitude of reasons. Professor of Philosophy Robert T. Carroll, Ph.D., gives a long list of these reasons on the online counterpart to his 2003 book, The Skeptic’s Dictionary:
Stories are prone to contamination by beliefs, later experiences, feedback, selective attention to details, and so on. Most stories get distorted in the telling and the retelling. Events get exaggerated. Time sequences get confused. Details get muddled. Memories are imperfect and selective; they are often filled in after the fact. People misinterpret their experiences. Experiences are conditioned by biases, memories, and beliefs, so people's perceptions might not be accurate. Most people aren't expecting to be deceived, so they may not be aware of deceptions that others might engage in. Some people make up stories. Some stories are delusions. Sometimes events are inappropriately deemed psychic simply because they seem improbable when they might not be that improbable after all. In short, anecdotes are inherently problematic and are usually impossible to test for accuracy.
Personal experience may seem like the most reliable of all tests, but people are more easily fooled than they would like to believe – ask any magician. More rigorous investigation has shown again and again that many so-called “folk remedies,” some of which have been in use for centuries, simply are not efficacious beyond the psychological phenomenon known as the placebo effect.
Wheatgrass juice is a case in point. According to the Natural Medicines Comprehensive Database, people today drink this concoction as a curative for a wide range of ailments, including cancer, preventing gray hair, kidney stones, bronchitis, arthritis, and to rid the body of “toxins” (a catch-all term that means nothing from a medical standpoint).
In an article for the August 2008 issue of Scientific American magazine titled “Wheatgrass Juice and Folk Medicine,” Dr. Michael Shermer, Adjunct Professor of Economics at Claremont Graduate University and founder of Skeptic magazine, traced the modern promotion of wheatgrass to a holistic health practitioner named Ann Wigmore, who began campaigning for the plant in the 1940’s after being inspired by a biblical tale of King Nebuchadnezzer, as well as the tendency of dogs and cats to eat grass to aid in digestion.
According to Shermer, Wigmore posited that wheatgrass had such palliative powers because of the enzymes and chlorophylls contained therein. Wigmore, however, completely misunderstood how digestion works. To explain why it makes no sense, Shermer quoted William T. Jarvis, retired professor of public health at the Loma Linda University School of Medicine and founder of the National Council against Health Fraud:
“Enzymes are complex protein molecules produced by living organisms exclusively for their own use in promoting chemical reactions. Orally ingested enzymes are digested in the stomach and have no enzymatic activity in the eater.” Jarvis adds, “The fact that grass-eating animals are not spared from cancer, despite their large intake of fresh chlorophyll, seems to have been lost on Wigmore. In fact, chlorophyll cannot ‘detoxify the body’ because it is not absorbed.”
Ann Wigmore. Courtesy of healthbanquet.com
Despite the obvious inanity of Wigmore’s reasoning, there continues to be a vibrant market for wheatgrass, perpetuated entirely by personal testimony and careful marketing.
So what is the big deal? If products like wheatgrass juice - or traditional therapies like acupuncture or homeopathy - make someone feel better, why should it matter if there are no well-designed clinical trials to support any significant effects? Why does an organization like the FDA refuse to let these remedies join the ranks of the approved? What is the harm?
The harm comes in three forms. The first is that anecdotal claims often rest on unstated (and sometimes stated) premises that fly in the face of basic scientific principles. For instance: 160 years ago, Samuel Hahnemann, the father of homeopathy, based his methodology on the concept that water has “memory,” and that diluting a substance renders it more powerful. The most “effective” homeopathic remedies are usually so diluted that they do not contain a single molecule of the original substance. People who accept these ideas not only become misinformed about medicine – they must reject most of physics, chemistry, and biology to maintain their belief. The diametric opposition to well-established facts becomes a huge barrier to understanding many other aspects of science, and undermines critical thinking abilities altogether.
The second harm is borne by those with ills that might be treated by standard methods of care, but whose conditions are worsened because they choose alternatives that have only anecdotal backing. The conflation of correlation with causation is a common culprit in such anecdotal misinterpretations. If, for instance, person X receives homeopathic treatment for prostate cancer and discovers that the cancer has gone into remission, he may decide that the treatment caused the remission. In doing so, he ignores the fact that cancers sometimes spontaneously subside, that there may be other factors that contributed to his newfound health (exercise, other treatments, and so on), and – most importantly – that persons A to W did not obtain the same results. While X champions his homeopathic cure, these other folk never get the chance to tell the stories of their failures, because they are dead.
This is more than just a hypothetical scenario. Homeopaths have claimed that their remedies can effectively treat diseases like malaria, gonorrhea, and even AIDS. One such practitioner named Peter Chappell peddles his cures (which are proprietary, because, he says, they are “ahead of the science and because the terms are not there, to explain it within science”) to the sick and desperate in Africa. Chappell does some fancy footwork to explain his method’s ineffectiveness in the developed world. In a 2007 interview for the online homeopathic forum Hpathy, he said the following of “White people and Europeans”:
“White people with AIDS frequently don't respond so well to PC AIDS [the name of his remedy] because they have a lot of other issues running at the same time, which are more important often than the AIDS, in that they depress the immune system far more than the virus.”
Peter Chappell. Courtesy of minimum.com
Chappell seems to forget that Africans suffer from much higher rates of most diseases than anyone else on the planet, that they have poorer nutritional health, that they frequently face stressful and unsanitary living conditions, and that some African countries have average life spans that are about half as long as those enjoyed in rich countries. Yet white people (whatever that means), we are told, have more issues depressing their immune systems. Is it at all more likely that “white people” would require clinical trials, while desperate African nations are more willing to accept the word of anyone who promises hope for their ailing populations? As it stands, it is impossible to tell, since Chappell has no statistical evidence to back his claims.
Most people who rely on anecdotally supported treatments are not on the verge of death, of course, and many folk remedies are applied only in relatively benign situations, such as the occasional scrape or cold. Yet even in these cases, the consumer of folk remedies faces a third potential harm: a loss of money and time. According to a December 2009 article in the Nutrition Business Journal, which describes itself as “a research, publishing and consulting company serving the nutrition, natural products and alternative health care industries,” U.S. consumer sales for the subcategory of supplements that includes homeopathic remedies, fish/animal oils, probiotics, and a few other supplements grew by $270 million in 2008 to $4.5 billion. This, according to the article, was a “less than stellar” year for the products. Such huge investments by the American public divert funding from therapies and treatments that have more than testimony to vouch for them.
Anecdotal evidence is clearly not enough to make a folk remedy worth relying upon. This does not mean that folk remedies carry absolutely no weight, however. Just as they steer people in wrong directions, they have been the source of a great number of legitimate discoveries. One famous example of just such a remedy involved British sailors in the 1700’s who suffered from scurvy, a condition caused by Vitamin C deficiency. Dr. James Lind conducted one the earliest clinical trials in history, and discovered that sailors who were given a regimen of two oranges and one lime a day improved. In time, lime juice was substituted for lemon juice, and the sailors who imbibed this everyday substance were dubbed “limeys.”
More recently, certain fields that were once the realm of the humanities have begun to standardize their practices by taking more evidence-based approaches to research. This has been especially true of clinical psychology, which has coupled itself with neurobiology and other “hard sciences” as well as evaluating its internal methods to establish greater standards of efficacy and accountability. Discussions of creating a new accreditation system have spawned fresh debates over just how much the field can benefit. In a December 4 segment on National Public Radio’s Science Friday titled “Can Science Make Psychotherapy More Effective,” Dianne Chambless, the Merriam term professor of Psychology at the University of Pennsylvania in Philadelphia, related how incorporating new evidence into her practice had improved outcomes in people with obsessive compulsive disorder:
…back in the 60s, early 70s, we were treating obsessive compulsive disorder with relaxation approaches because that seemed to help other anxiety disorders. And we weren't really getting anywhere.
And then in England people developed a new treatment called exposure and ritual prevention. And we read about this. I was in training. And we started with great excitement to try this new treatment, and it worked. So we weren't different as therapists. We were still the same caring people that we used to be. But we tried a new treatment that was based on the science of fear and learning. It worked. It meant some people who were crippled by their problems became functional people, and around 70 percent of them got very much better.
Chambless went on to say that despite the obvious difference between the two approaches, surveys of clinical psychologists have shown that more continue to use the defunct relaxation techniques. Both types of therapy became widespread because of anecdotal evidence. Further studies showed one to be more effective than the other. This, she believes, is just one reason to bring psychotherapeutic methods further beyond the realm of anecdote.
The problems with relying on remedies that are not supported by scientific evidence are multitudinous. This makes the need for an approval process for medicines obvious. However, no system is perfect. As the primary body for determining the safety and efficacy of medicines in the U.S., the FDA bears the weight of criticisms, some of which are legitimate, from all sides. As with any governmental organization, politics and money play a role in its function. Moreover, the very nature of the clinical trial may impede many remedies from being approved – or even reviewed.
The Mayo Clinic’s website offers a few reasons why some alternative medicines never make the cut for approval by the FDA. Among other things, it points out that “large, carefully controlled medical studies are costly. Trials for conventional medications or procedures are often directly or indirectly funded by the government or drug companies, giving conventional treatments more resources to do studies.”
When there are no lucrative patents to be gained, for-profit companies are less likely to pursue research into folk medicines. This lack of funding has historically kept many potentially beneficial remedies from being investigated.
As the alternative medicine industry has gained ground in popular culture, however, the government has taken an active role in funding studies through the branch of the National Institutes of Health called the National Center for Complementary and Alternative Medicine (NCCAM). In 2008, approximately $300 million (around 1%) of funding for medical research went to this institution.
Graphic Courtesy of nccam.nih.gov
Clinical trials take years to conduct, and this poses another barrier to alternative modalities. Many of them make so many claims that investigating all of them would take decades. Some of these claims are so outrageous that scientists do not deem them worthy of investigation whatsoever, but others simply have not been subjected to enough research for the FDA to make reasonable decisions about them.
When to stop research is another open question. If a substance or therapy has shown no effect for a given ailment in hundreds of well-designed trials, does one pursue research into another, slightly different, use? How much time, money, and effort does any modality deserve? The decision to end a line of inquiry is partially dependent on resource prioritization. When the results of research keep coming back negative, other pursuits must eventually take precedence.
Although it may take a long time for a treatment to gain acceptance under the current regulatory regime, the complaints of some practitioners whose methods do not hold up even after being scrutinized are rarely justified. They have careers – and often personal convictions – at stake, and may resort to special pleading to defend their failures. This is exactly what happened in 2005, when the medical journal The Lancet published the most thorough analysis of homeopathy ever conducted, synthesizing data from over 100 trials related to numerous illnesses and found that homeopathy had no better results than placebo. Not only did homeopaths unanimously decry the research; at least one group went as far as to say that placebo-controlled randomized trials were not a proper method for testing their wares:
Two primary concerns for homeopaths are that the treatment is holistic and that it is individualised. Treatment cannot be standardised and patient response is unpredictable. RCTs are looking for specific effects whereas homeopathy is attempting to produce general health effects as well as specific effects – homeopathy treats the whole person….Patients may choose homeopathy or acupuncture or other forms of complementary and alternative medicine because of preferences and perhaps even the potential for responding. Randomized controlled trials do not necessarily recruit such patients – they take patients with a conventional diagnosis, but do not screen for patients who typically may gravitate to homeopathy or other types of complementary and alternative medicine.
In other words, the only proof that these homeopaths can accept is anecdotal – and then, only if it confirms their beliefs. And that is exactly why bodies like the FDA must continue to exist – to divide the purely anecdotal from the thoroughly researched. At its best, that fallible but invaluable service is all we can – or should - expect from it.
References:
1. Carroll, Robert T. “Anecdotal (testimonial) evidence.” The Skepdic’s Dictionary. 1994-2009. 1 Dec. 2009http://www.skeptic.com/testimon.html
2. “Wheatgrass.” Natural Medicines Comprehensive Database. 1995-2009. Therapeutic Research Faculty. 1 Dec. 2009 http://www.naturaldatabase.com/(S(bvtcrabgv0exbxmhw4kbgyar))/nd/Search.aspx?cs=&s=ND&pt=100&id=1073&fs=ND&searchid=18119729
3. Shermer, Michael. “Wheatgrass Juice and Folk Medicine.” Scientific American Magazine. August 2008. 1 Dec. 2009. http://www.scientificamerican.com/article.cfm?id=how-anecdotal-evidence-can-undermine-scientific-results
4. ibid.
5. Baum, Michael. “Should We Maintain an Open Mind about Homeopathy?” The American Journal of Medicine. Volume 122, Issue 11, Pages 973-974. November 2009. Dec. 2, 2009. <http://www.amjmed.com/article/PIIS0002934309005336/fulltext>
6. Alan V. Schmukler. “Interviews: Peter Chappell.” Hpathy e-zine. August 2007. Dec. 1, 2009. http://74.125.93.132/search?q=cache:RHl2oBsaqqwJ:www.hpathy.com/interviews/peterchappell.asp+PC-remedies+aids&cd=6&hl=en&ct=clnk&gl=us
7. “About Us.” Nutrition Business Journal. 2009. Dec. 5, 2009. http://nutritionbusinessjournal.com/aboutus/
8. “Supplements Stand Out As 2008 Sales Bright Spot for U.S. Nutrition Industry.” Nutrition Business Journal. Dec. 4, 2009. Dec. 5, 2009. http://74.125.93.132/search?q=cache:6Z11VcfRn2QJ:subscribers.nutritionbusinessjournal.com/supplements/0601-supplements-bright-spot/+homeopathic+industry+profits+2008&cd=7&hl=en&ct=clnk&gl=us
9. Corte, Corinne. “Why are English Sailors Called Limeys?” Ask a Biologist. Arizona State University. Dec. 1, 2009. http://askabiologist.asu.edu/research/scurvy/index.html
10. “Can Science Make Psychotherapy More Effective?” Science Friday. NPR. Dec. 4, 2009. http://www.npr.org/templates/transcript/transcript.php?storyId=121092295
11. “Complementary and Alternative Medicine – What is it?” Mayo Clinic. Mayo Foundation for Medical Education and Research. Oct. 24, 2009. Dec. 2, 2009. http://www.mayoclinic.com/health/alternative-medicine/PN00001/NSECTIONGROUP=2
12. Noble, Rob. “Alternative, complementary and traditional medicine and HIV.” AVERT. Nov. 12, 2009. Dec. 2, 2009. http://www.avert.org/alternative-medicine-hiv.htm
13. Shang A. et al. “Are the clinical effects of homoeopathy placebo effects? Comparative study of placebo-controlled trials of homoeopathy and allopathy.” Lancet. Aug. 27, 2005. 366(9487). Dec. 2, 2009. http://www.ncbi.nlm.nih.gov/pubmed/16125589
14. Chatfield K. and Relton C. “Are the clinical effects of homeopathy placebo effects? - A full critique of the article by Shang et al.” European Central Council of Homeopaths. Sep. 2005. Dec. 2, 2009. http://www.ncbi.nlm.nih.gov/pubmed/16125589
