We are programmed to respond to testimonials. Blame evolution. For most of human history, we had only two ways to learn about the world: our own observations and what other people told us about their experiences. Gathering information from others gave us a survival advantage. Unfortunately, those testimonials can be unreliable. We now have something far more reliable: science. But science doesn’t come naturally. We have to learn to trust studies over stories, and not everyone learns.

Proponents of complementary and alternative medicine (CAM) rely heavily on testimonials. They think that’s all they need. If people say a treatment helped them, that’s all the “evidence” they need or want. But testimonials are not evidence. Ten testimonials are no better than one; a thousand testimonials are no better than ten. Testimonials are useful only because they can indicate that an untested treatment is promising and worthy of formal scientific study. CAM practitioners do not value science and often don’t understand it. But they are increasingly aware that others do value science, so more and more, they are trying to provide scientific evidence. Their efforts are often misguided and laughable, and they often end in failure.

When I visit a CAM website, I look for a “scientific evidence” tab. Sometimes the people pushing CAM don’t even pretend to have any supporting science. When they do, the studies they cite are all too often poor science, junk science, or irrelevant. Sometimes they even disprove the claims they are alleged to support. What I find most disturbing is when a tab for “evidence” leads only to a series of testimonials. They know how convincing testimonials can be, and they don’t understand why testimonials don’t count as evidence.

How do we know a testimonial isn’t bogus? Does the person even exist? If it is a real testimonial, how do we know the person wasn’t misled for one of the many reasons people can erroneously believe that a treatment has helped them when it hasn’t? The late Barry Beyerstein listed seven of these reasons in his classic article “Why Bogus Therapies Often Seem to Work.”

1. The disease may have run its natural course.
2. Many diseases are cyclical.
3. The placebo effect may be responsible.
4. People who hedge their bets credit the wrong thing.
5. The original diagnosis or prognosis may have been incorrect.
6. Temporary mood improvement can be confused with cure.
7. Psychological needs can distort what people perceive and do.

There are more than just these seven. Another source of error is our notoriously fallible memory. Fortunately, these pitfalls can be avoided by validating the treatment with scientific studies that control for confounding factors such as placebo responses, compliance effects, and judgmental errors. 

Kailo is a pain relief patch based on laughable energy medicine quackery. Its advertising brags about the results of “An IRB-approved study”—as if “IRB-approved” meant something special. No, all studies on humans require approval by an Institutional Review Board to protect the rights of research subjects. Not only that, but the results its manufacturers describe are only preliminary results. The study is meaningless because it lacked a control group. Participants simply used the Kailo patch, and a large percentage of them reported that their pain improved. No surprise there. Pain is subjective, and suggestion is powerful: pretty much anything you offer will elicit reports of improvement. 

That wasn’t a legitimate study; it was more like a group of testimonials. CAM supporters love testimonials because they convince customers and are easy to obtain. Every snake oil peddler and charlatan can provide testimonials for their wares. 

CAM is all about testimonials. I don’t think there’s any way we could convince its supporters that testimonials are not evidence. If they really “got” science and the need for real evidence, they could no longer believe in the effectiveness of CAM. Thank goodness some of us know better.