Skepticism and Pseudoexperiments

Benjamin Radford

From: Volume 44, No. 5
September / October 2020

Pseudoscience is something that presents as, or is mistaken for, science—but is not. Pseudosciences are especially resilient and pernicious because they benefit from the hard-earned legitimacy of real science as arbiter of objective truth. From phrenology to astrology, homeopathy to facilitated communication, pseudosciences gain believers conditioned to (rightfully) respect science and its methods.

Pseudoexperiments are similar in that they typically present an experiment—a crucial tool in science—and apply it in a popular culture context (though without the input of actual scientists, who might needlessly complicate the attempt with seemingly frivolous trivialities such as control groups, blinding, research design, or, really, anything that might make it valid).

Pseudoexperiments are common in everyday life, though rarely identified as such. More often they’re seen as rhetorical devices, entertainment, or simply news. They have a populist plausibility because they have a superficially valid lesson: To understand some phenomenon, you don’t need to look at research design or boring statistics. You can “see for yourself” what’s going on, what the truth is. They are especially effective because they give the illusion of objectivity. Pseudoexperiments, like scientific experiments, are usually portrayed as unbiased—though often activist—“real world” examples to prove a point. Like classic pseudoscientists, pseudoexperimenters want the prestige of science but don’t want to put in the effort to make it meaningful or valid. Their goal is not the discovery of nuanced truth about the world but instead promoting an agenda by means of a simplified, memorable “gotcha” moment.

A pseudoexperiment is not necessarily the same as a flawed experiment. Conducting a perfect scientific experiment is very difficult (if not impossible), which is why p-values are factored into a study’s results. Flawed research is a hallmark of many pseudosciences and paranormal topics, from ghosts to crop circles to cryptozoology. For pseudosciences such as psi, there is an inverse correlation between the quality of the research and the amount of good evidence obtained; the better the controls, the less empirical evidence appears (as Jim Alcock, Stuart Vyse, Scott Lilienfeld, and others have noted in this magazine). There are several types of pseudoexperiments.

Pseudoexperiments as Television Fodder

Most pseudoexperiments are done for a viewing audience, often for journalistic, entertainment, or commercial purposes. In January 2006, security specialist Bill Stanton appeared on The Today Show posing an alarming and provocative question: If you saw an innocent child being kidnapped by a stranger, would you help?

There are several approaches that the producers could take to answer the question, including interviewing a social psychologist about research on bystander effects. But the producers chose a more dramatic angle: using hidden cameras to record a seven-year-old named Rachelle faking being abducted on a city street. Rachelle’s mother watched from a surveillance van as Stanton approached the girl, who stood alone in the middle of a sidewalk playing a video game. Stanton walked up to Rachelle and took her by the arm, saying things such as, “There you are, young lady! You come with me,” while Rachelle protested, “No, no … you’re not my daddy!”

Stanton and Rachelle repeated the scenario several times, and rarely did bystanders intervene. Some kept walking, and others glanced briefly at the scene, but few approached. The Today Show anchors called the results “shocking,” and to everyone involved (and probably the audience), this seemed a clear and sad case of people reluctant to help someone in need. “It’s frightening that no one will help,” Rachelle’s mother said.

Yet there may be a good reason more people didn’t get involved—one completely missed (or ignored) by Stanton and the Today Show producers: the bystanders didn’t believe that they were actually seeing a child abduction. Because the “abduction” was poorly staged, it’s more likely that those who witnessed the scene simply—and accurately—recognized that the child was not in real danger. From the footage, it’s clear that Rachelle was not an actress and didn’t act scared when Stanton approached her. Her protests sounded like a typical child’s whines instead of panicked pleas for help. Stanton did not strike the child or hurt her in any way, and Rachelle didn’t scream, kick at, or fight off the man supposedly trying to abduct her. In short, there was little that would convince the average person that she was genuinely in danger. The problem isn’t the seven-year-old’s acting; it’s a poorly conducted pseudoexperiment by Stanton and NBC News. The Today Show’s hidden camera test would be valid only if the bystanders actually believed that the child was in danger; it’s not clear that was the case.

The highly rated ABC News series What Would You Do?, hosted by John Quiñones, follows an identical format and is now in its fourteenth season. Hidden cameras record people in public places such as restaurants and involve social justice issues, including race, gender identity, disability, and parenting. Will strangers step in to confront a staged racist or sexist remark made between two actors? Will passersby act against an able-bodied person using a handicapped spot? Quiñones, who was inspired to go into journalism by Geraldo Rivera—himself no stranger to alarmist television, especially during the “Satanic Panic” scare of the 1980s—said that he believes his show “holds up a mirror” to America.

While some of these staged performances are inevitably better executed than others, it’s not clear how valid or generalizable the results are. People may choose (or choose not to) publicly intervene in events nearby for any number of reasons unrelated to the hypothesis offered by the show (apathy, prejudice, etc.). Bystanders may not fully understand what’s going on; they may have visual or auditory disabilities and sincerely not have noticed the seemingly alarming event; they may be socially anxious or fearful about confronting others; and so on—though often the broadcast reactions are precisely what the producers hoped they’d be. Like Candid Camera and countless other shows, these pseudoexperiments are voyeuristic entertainment, not science experiments or philosophical musings on responsibility such as seen in Rashomon (1950) or Force Majeure (2014).

In some cases, it’s a physical pseudoexperiment. In 1992, Dateline NBC aired a program titled “Waiting to Explode,” which examined claims that General Motors trucks exploded when involved in low-speed collisions due to faulty gas tank design. To demonstrate this, the program showed footage of a truck exploding when being hit from the side, ostensibly as part of a safety test. It was later revealed that the explosion had been staged. Not only had the truck’s gas cap been intentionally loosened, but an explosive charge had ignited the fireball. GM sued the network, and the journalistic credibility of NBC News suffered for years because of this pseudoexperiment.

Pseudoexperiments as Social Activism

While ratings-driven journalism is a common source of pseudoexperiments, many laypeople use them to raise awareness of social issues. For example, in 2018 a San Antonio woman named Stacey Alderete recorded herself entering her child’s school. She had just dropped her daughter off and returned to the school after grabbing a backpack. She’s seen in a short cell phone video following others through the main entrance and walking through a hallway. She’s seen by several people but not confronted by any security staff.

To Alderete, it was a bold experiment that exposed a glaring—and possibly deadly—flaw in school security, one that could be exploited by a would-be mass shooter. The day after posting the video to YouTube, Alderete appeared at a public school-board meeting and said, “I proved yesterday just how vulnerable to violence or school shootings our children are at the high school. My video speaks volumes.” She later added, “If I was able to do this without being detected, ANYONE CAN!” That interpretation was embraced and widely shared on news and social media (Bates 2018).

A more parsimonious explanation is that she was recognized. Alderete did not slip into the school unnoticed with a suspicious item; she was seen and (correctly) recognized as a nonthreatening mother of a child attending there. A young male carrying a gun, or even a large backpack, would arouse more fear and suspicion than an adult woman who’s been seen at the school many times before. The mere fact that she wasn’t personally confronted by security, of course, doesn’t mean that her presence was unnoticed. It’s entirely possible that she was seen on surveillance cameras by security staff who recognized her as having dropped off her daughter earlier and assumed she was returning to give her child a forgotten item or message.

It was later revealed that Alderete had an ulterior motive for her pseudoexperiment; it wasn’t conducted to objectively test school security but instead because she had lost a re-election campaign as a school board member, emphasizing security. “Ms. Alderete is simply a disgruntled former board member that lost her last election and has spent a lot of her time since then trying to create havoc for the school district,” Superintendent Abelardo Saavedra said. He also noted that it was “totally ridiculous” for Alderete to claim that she had exposed security flaws, because as a former school board member, she is a “fairly high-profile parent” and widely recognized by school staff and security (Caruba 2018).

Celestia Ward, in an article on “outrage vampires,” described a typical social media pseudoexperiment by YouTube influencer “Joey Salads” and his 2016 viral video exposing the scorn that breastfeeding women face:

The so-called experiment contrasts a sexy woman in a low-cut top with a young breastfeeding mom, both sitting on a mall bench “for science!” We see the attractive model receive plenty of glances, and one fellow even sits down and hits on her. Then, after the title “Now lets see how people react to Breastfeeding” [sic] flashes, we see the young mother heckled over and over again. In this two-minute video, four separate people hurl the same word, “disgusting,” right at her, and each time she meekly apologizes “I’m sorry, he’s hungry.” All these disgusted mall denizens also had an approach I’ve never seen in real-life situations. Rather than just grumbling or expressing their distaste in whispers as they amble by, each person walks right up to the young mother, stands in a position easily captured by the camera, and delivers the insult clearly and loudly … as if they had been coached on what to do. It was only one of many similar faked videos by Salads, who, when asked, “Why do you feel the need to fake an experiment?” replied he was “just trying to make something shocking and create controversy.” (Ward 2018)

Other similar stunts include videos by a man named Adam Saleh, who claimed he was removed from a Delta Airlines flight because he spoke Arabic and was accused of being a terrorist. Like Alderete, he recorded the incident and posted it online, generating news headlines and anti-racist outrage. However, other passengers disputed Saleh’s claims, reporting that he had lied: Saleh was not merely speaking Arabic but shouting on the plane, being disruptive and confrontational—clearly hoping to provoke a response from ordinary passengers that he could falsely portray as racism that would be sensational enough to go viral. In fact, “Saleh is a serial prankster who frequently makes YouTube videos in which he pulls stunts on flights. His YouTube channel features films in which he conducts ‘experiments’ to determine how passengers will respond to travelling alongside Muslims and people speaking Arabic” (Boult 2016).

Another social justice pseudoexperiment made national headlines in 2014 when a viral video showed a woman walking along New York streets with some men catcalling behind her. The video, filmed and released by an organization called Hollaback, initially garnered widespread attention (with over 40 million views) and praise, hailed as an important social experiment revealing the scope of street harassment and catcalling. But it soon attracted controversy when its status as a pseudoexperiment was revealed.

People began taking a closer look at what the video did—and didn’t—show, what it revealed, and what it concealed. Nearly all the men seen catcalling, whistling, and making propositions at the white, conventionally attractive actress were people of color, mostly African American men. The video was clearly edited; she allegedly walked for nearly ten hours, but the video was just two minutes long. The footage wasn’t merely sped up; instead only a tiny percentage of it was seen. Few suspected that it was outright staged (as Joey Salads’s and Adam Saleh’s videos had done), but this raised the question: Was the segment shown really representative of her experience during the walk, or did the editors cut it to make it seem that blacks were disproportionately harassing the woman? Part of that answer was related to where she walked and when.

The social justice video itself soon smacked of racism, and experts took a closer look. Professor Zeynep Tufekci (2014) analyzed the video, noting:

I’ve taught Introduction to Research Methods to undergraduate students for many years, and they would sometimes ask me why they should care about all this “method stuff,” besides having a required class for a sociology major out of the way. I would always tell them, without understanding research methods, you cannot understand how to judge what you see. The Hollaback video shows us exactly why.

Tufekci’s excellent analysis is too complex to treat here; see her article “Hollaback and Why Everyone Needs Better Research Methods.”

Advertising Pseudoexperiments

Pseudoexperiments have been a staple of advertising for generations. Countless side-by-side comparisons have “proven” how much better the hero product is versus the leading brand of everything from removers for stubborn laundry stains to car waxes—while leaving out important caveats and differences between the two.

In recent years, companies have used social media to share advertising pseudoexperiments, often in the hope they will go viral, amplifying their reach. One famous example is a series of short videos created by Dove/Unilever, a 2013 campaign called Real Beauty Sketches. In it,

Dove hired former police forensic artist Gil Zamora to illustrate some psychologically revealing sketches. In a campaign created by Ogilvy Brazil, a series of women described themselves to Zamora in minute detail, from behind a curtain. The artist in turn created composites as though trying to identify a criminal. Next, each participant was asked to describe another woman present. The results are dramatic and sort of moving. Viewing the two sketches side by side—one based on self-description, one from a friendly stranger—it’s clear how unflattering the women’s own self-assessments are. (Berkowitz 2013)

The campaign was enormously successful, generating nearly 70 million views on YouTube and praise as empowering and poignant from news media, media critics, and feminist organizations. Soon critics noted flaws in the pseudoexperiment, suggesting that the advertisement didn’t quite “prove” anything.

For one thing, the positive descriptions of the women in the video were rooted in what’s conventionally attractive: thin, mostly Caucasian, with blue eyes and smooth complexions. Whatever the intended message, the focus was still on appearance (which is to be expected from a cosmetics company). From a research design point of view, it’s not clear why Dove assumed that self-descriptions would (or should) match descriptions by other people. This is due in part to demand characteristics: people tend to respond in socially appropriate ways that make them appear kind and positive. People—especially women—tend to err on the side of humility when describing themselves to others, so as not to appear vain. As Autumn Whitefield-Madrano wrote in her book Face Value: The Hidden Ways Beauty Shapes Women’s Lives, “We [women] learn that women aren’t supposed to refer to themselves with words bolder than reasonably attractive” lest they be seen as vain (Whitefield-Madrano 2016, 58).

Similarly, people tend to be kind when describing the appearance of others so as not to appear cruel or mean-spirited. When being asked by an interviewer—on camera no less—to describe another person’s appearance, few of us would be eager to highlight their flaws. The sketch artist was also aware of the pseudoexperiment’s purpose and may have consciously or otherwise guided his depictions according to the client’s goals.

Thought Pseudoexperiments

Of course, not all experiments are done in the real world. Philosophers, social scientists, and others engage in what are called thought experiments, whereby using various premises, assumptions, and rules of logic they can imagine what would, or should, happen given certain premises or conditions. They are usually framed as a rhetorical question intended to expose a real or imagined hypocrisy or double standard. Thought pseudoexperiments are common on social media, in which, for example, a meme (often rhetorically) asks the reader to compare two people or situations and typically directs them to (an apparently self-evident) conclusion. These thought pseudoexperiments typically involve political, environmental, or social justice issues and are framed to provoke outrage or righteous indignation.

Examples include memes such as “Imagine if [Harvey] Weinstein was black” under a photo of Bill Cosby’s mugshot, suggesting that if the Hollywood producer were black, he would already be arrested and convicted, even though the cases are quite different. There are countless variables in the two cases. Race is certainly one of them, but there’s no reason to think it’s the only—or even the most important—one. It’s not a real experiment; it’s just asking us if we can imagine person A swapped out for person B in situation X. Unless you have the imagination of a turnip, the answer is always yes. We can imagine a scenario in which the outcome is as described. But that doesn’t mean it is likely or even probable (for other examples, see Radford 2019). Similar memes compare two newsworthy people who got unusually long (or short) sentences for (allegedly) similar crimes, but from different socioeconomic strata. It’s rhetoric thinly disguised as a neutral, objective question, akin to a conspiracy theorist “just asking questions.” But if those premises are biased, skewed, or wrong, then the conclusions drawn from them will often be flawed.

Experiments versus Pseudoexperiments

Flawed experiments are common, but they may not have the same potential to mislead that pseudoexperiments do. From skeptical, media literacy, and critical thinking points of view, there are several flaws inherent in pseudoexperiments.

Perhaps most obviously, they emphasize anecdote over data, and anecdotes about a social problem are not necessarily valid evidence about that topic. The question is not what happened to one person in one specific case but instead whether that case is representative of the larger group or population. Pseudoexperiments always have an obvious agenda, so in that regard it’s not an experiment at all. In scientific experiments, manipulating the data is a grave transgression; in pseudoexperiments, manipulating the data is the whole point. There’s also a filedrawer effect: if a person conducts a pseudoexperiment and it doesn’t turn out how they wanted it to, nobody sees it—or they simply edit out the inconvenient bits.

A second fallacy is failing to consider alternative explanations. Legal outcomes in court cases are especially vulnerable to misrepresentation in pseudoexperiments. People can cherry-pick examples all day to suit their needs. If you want to find a case where a wealthy white male was accused of something horrible and “got away with it” (whatever that might mean in a given context—from charges being dropped to an acquittal at trial, a mistrial to probation) presumably due to status and privilege, you can do it. The same is true if you swap out any of the variables: rich or poor, male or female, gay or straight, black or white, and so on. In mere minutes, an internet search will turn up some case, somewhere, in which a given category of person did (or didn’t) “get away” with some presumably outrageous crime or act, plausibly due to whichever factor the presenter wishes to highlight. Often the cases are notable and newsworthy precisely because they’re exceptional and unusual. Ordinary people doing ordinary things—or receiving statistically average sentences for ordinary crimes—rarely make the news. Selection and confirmation biases direct us to the dramatic, unrepresentative outliers and make inherently flawed examples for comparison.

Pseudoexperiments point the audience to a single, seemingly self-evident, overriding factor to the exclusion of others. The factor being highlighted could certainly be the main reason for the disparity, or it may have played a part, or it could have been wholly irrelevant. There’s usually no way to know without a much closer look at research and statistics. In many cases, the overall point or “larger truth” of the meme or experiment is valid, but the pseudoexperiment chosen to prove or demonstrate that claim is not.

Pseudoexperiments are powerful because they’re simple, easily digested, and effective at pushing buttons and provoking anger and outrage. They can also cause harm. The examples discussed here may make parents and children feel less safe in schools and in public; make Arab Americans feel less welcome on passenger airlines (or more self-conscious about speaking Arabic); make nursing mothers feel uncomfortable breastfeeding their babies in public; reinforce racist stereotypes about catcalling black men; and so on. Some of these are attention-seeking hoaxes, while others are misguided social justice stunts. Regardless of the methods and motivations, pseudoexperiments mislead and misinform. The real world is complex and hard enough to understand without pseudoscientific pseudoexperiments.

 

Acknowledgment

Thanks to Susan Gerbic for suggesting this article.

References

Benjamin Radford

Benjamin Radford, M.Ed., is a scientific paranormal investigator, a research fellow at the Committee for Skeptical Inquiry, deputy editor of the Skeptical Inquirer, and author, co-author, contributor, or editor of twenty books and over a thousand articles on skepticism, critical thinking, and science literacy. His newest book is America the Fearful.


This article is available to subscribers only.
Subscribe now or log in to read this article.