Cover Image Source: Twitter

 

---

Imagine the following scenario. Two social scientists write an article proposing a provocative new theory about women’s values, preferences, and the very essence of what it means to be a woman. The authors make some logical arguments, cite publicly available data, and, in support of each point, include several quotes—let’s say forty quotes altogether—from published works by a variety of women. The article is submitted to a prestigious journal, goes through the peer review process, and is accepted for publication. However, after the article is published online—but before it appears in print—another scholar points out that at least ten of the forty quotes are not from reliable sources and are very likely written by men, not women. What should happen to the article?

I can imagine three possible courses of action, although there are many others:

1. Nothing happens. The article is allowed to stand as originally published with the voices of men substituted for those of women in some cases.
2. Remove suspect quotes. If convincing evidence is presented that the ten quotes are unreliable, ask the authors to remove those quotes from the article. The quotes were merely supporting data, and given that the authors have other evidence and many remaining quotes to back up their arguments, a corrected version of the article could be allowed to stand.
3. Retract the article. This most severe action of pulling the article completely might be justified if the authors knew about the controversial nature of the sources and did not acknowledge it in the article, which seems unethical at best and fraudulent at worst.

The foregoing is not just a thought experiment. Recently something very like it happened. The arguments being made were about the nature of autism, not women, and I and eight other colleagues played the role of the scholar pointing out the error in the accepted article. Unfortunately—in my view—the eventual outcome was very close to Nothing happens. As a result, I believe this episode has important implications for the status of the peer review process and the creeping acceptability of pseudoscience. But I am getting ahead of myself. Here’s what happened.

Once upon a Saturday Night

One Saturday night in July 2018, I was at home alone aimlessly poking at my computer when an eye-catching GIF of a cat furiously typing on a laptop came across my screen. The GIF was attached to a tweet sent from the account of the journal Behavioral and Brain Sciences encouraging authors to write commentaries to accompany an article that had been accepted for publication.

Behavioral and Brain Sciences (BBS) is a prestigious journal published by Cambridge University Press in a unique open commentary format. The journal accepts target articles that are typically relatively long and substantive analyses of some issue in cognitive science, psychology, or neuroscience. Once the article is accepted, other scholars are invited to write short commentaries on the target article. Finally, once all the commentaries have been received and accepted for publication, the authors of the target article write a response to the commentaries. The resulting publication—target article plus commentaries plus responses to commentaries—is often quite long, and the journal’s format allows for an unusually deep exploration of innovative ideas. The list of people who have published target articles includes B.F. Skinner, Steven Pinker, Noam Chomsky, Daniel Dennett, and Keith Stanovich.

On the Saturday night I saw the cat GIF, I noticed that the target article was “Being vs. Appearing Socially Uninterested: Challenging Assumptions about Social Motivation in Autism” by Vikram K. Jaswal and Nameera Akhtar (J&A) (2019a). I did not recognize the authors, but because the title was intriguing and I had recently written about autism for Skeptical Inquirer, I clicked on the link.

What I found was quite a surprise. The authors proposed that the lack of social engagement shown by people with autism was not indicative of their desires. People with autism want to be more social—they simply face different barriers to expressing social behavior than neurotypical people. I had my own thoughts about the authors’ thesis, but that’s not what I found surprising. In support of their theories, J&A quoted from published articles and books written by people with autism. Naturally, they cited the authors by name, and although I did not know all of them, I recognized a few: Ido Kedar, Tito Mukhopadhyay, and Sue Rubin. Several of the quotes J&A used were from well-known users of either the discredited communication technique facilitated communication (FC) or its more recent variant rapid prompting method (RPM).

As readers of this column are aware, FC is a communication method based on the theory that rather than being cognitively impaired, many people with autism are not physically capable of making the movements required for communication. FC involves the steadying hand of another person—a facilitator or communication partner—placed on a hand, arm, or other body part of the nonspeaking person as they type with one finger on a keyboard. When used with nonspeaking individuals with autism, tests of FC have repeatedly shown that the words typed out are authored by the communication partner, not by the person with autism (Mostert 2001; Mostert 2010). Many double-blind studies have demonstrated with remarkable consistency that when asked simple questions, the nonspeaking person produces the correct answer only when the communication partner also knows the answer. At least twenty professional, educational, and advocacy organizations worldwide, including the American Psychological Association, the American Pediatric Association, and the American Speech-Language-Hearing Association, have issued policy statements against the use of FC (Behavior Analysis Association of Michigan, n.d.).

YouTube video of a group of people with autism using full hand-over-hand facilitated communication (also known as supported typing).

The newer method, RPM, involves the nonspeaking person pointing at a letter board that is held in the air by the communication partner. As is often the case with FC, the verbally fluent RPM communication partner reads out what the nonspeaking person has typed. The proponents of RPM have so far avoided testing for authorship, but the similarity of RPM to FC has prompted the American Speech-Language-Hearing Association (ASHA) to issue a statement discouraging its use, stating “information obtained through the use of RPM should not be assumed to be the communication of the person with a disability” (American Speech-Language-Hearing Association 2018).

Video of a man with autism using rapid prompting method with his communication partner (Washington Post video).

I was surprised that J&A used quotations from these well-known users of FC and RPM, but I was also concerned that they said nothing about how those quotes were obtained. As a result, a statement attributed to a book coauthored by Barb Rentenbach—who is nonspeaking—and her FC facilitator, was placed on equal footing with one by Gillan Drew, who is a very fluent speaker. It seemed extremely unlikely that the authors—experts in autism—were unaware of the criticisms of FC and RPM, and as a result, the lack of any mention of how these quotations were obtained seemed like a breach of ethics to me.

I was somewhat shocked by the boldness of J&A’s move. I imagined many peer reviewers and potential commentary writers would recognize some of the same FC and RPM users I had. For example, Sue Rubin, who attended Whittier College with her FC facilitator and was awarded a B.A. in Latin American history, was the star of the Academy Award nominated film Autism Is a World (“About Me,” n.d.). She is a genuinely famous FC user. But somehow the manuscript had gotten through peer review.

My first step was to email the editor of BBS that very Saturday night. I knew the editor a little and had recently served as a peer reviewer on a different manuscript submitted to the journal. I acknowledged that mine might be an annoying email to receive, but I had serious concerns about the article. The editor wrote back the next day to say that the article had been peer reviewed by experts, revised, resubmitted, and finally accepted for publication. The editor did not dismiss my concerns but said this was not a case where retraction was warranted.

I had not actually mentioned retraction in my email. Rather, I suggested the article should not have been published in its current form—something akin to option 2 above. Nonetheless, the editor said my issues with the manuscript might make a very good commentary on the J&A article and encouraged me to submit one. Although submitting a commentary seemed like a rather weak response to the problem, I decided it was better than not writing one and potentially letting the use of suspect quotes go undetected. But I felt like I needed backup. I sent a quick email explaining the situation to a small group of academics who are working to discourage the use of FC and RPM (see Vyse 2018) and asked them if they would be willing to sign on as coauthors of a commentary to accompany the J&A article. To my gratification, within twelve hours I had positive responses from eight people willing to join the effort: Bronwyn Hemsley, Russell Lang, Scott O. Lilienfeld, Mark P. Mostert, Henry D. Schlinger Jr., Howard C. Shane, Mark Sherry, and James T. Todd.

As it turns out, the process of getting a commentary accepted in BBS is fairly onerous. The first step is to write a very brief proposal—basically an abstract of what you intend to say in the commentary. This proposal goes through a first-round of review and can be rejected at this early stage. If accepted, you go on to write the commentary, which is limited to 1,000 words, and submit it for review. Again, a commentary can be rejected at this second review point, but if accepted, it eventually appears following the printed version of the target article.

By the time I had decided to take on this project and had acquired my coauthors, the deadline for the short commentary proposal was approximately twenty-four hours away. Looking a bit like the cat in the GIF, I quickly pounded out a proposal with the help of those colleagues who were able to respond on such short notice. Before long, we heard that we had made it through the first hurdle and were given the okay to write our commentary.

To prepare for the commentary, we entered every quote used in J&A’s target article into a spreadsheet. There were forty quotes in total. Many of the quotes were obviously from individuals who had speech, but in cases where people appeared not to use speech, we did our best to track down the sources cited. Together we made our best determination of the methods used to obtain the quotes, which, in a few cases, was impossible given the publicly available information. Ultimately, we judged that at least ten of the forty quotes were suspect. We labeled seven of the quotes FC or RPM, and three we identified as “likely FC.” Having completed the spreadsheet, we posted it and a brief introduction to our analysis on the Open Science Framework where anyone could look at them, and we invited discussion of our findings. The link to the files is here. As of this writing, no one has commented on our analysis.

It Gets Worse

When we looked into the quotes, the situation got a little worse. One of the quotes of people with autism was cited as “(Gralow 2008).” As it turned out, Christine Gralow was a special education teacher who worked with children with autism, and the source that J&A cited was an op-ed she wrote for the New York Times online. As far as we could tell, Gralow did not identify as being on the autism spectrum, and when we checked the source, it became clear that the quote that J&A attributed to her was not something she said. A Google search using the quoted phrase revealed that it actually came from a commenter, “jaymee,” on the New York Times website, but, of course, because it is the internet, there is no way to know who “jaymee” is. The authors, who one would imagine knew what they were doing, simply attributed the quote to Gralow and did not reveal the actual source.

We described this as an error in our spreadsheet accounting of the quotes, but because our space was limited and this error was not central to our critique of the use of FC and RPM, we did not mention it in our commentary. I did, however, inform the BBS editor of this mistake in a separate email sent at the time we submitted our commentary. Nonetheless, the quote remains in the printed version of the article, with the original incorrect citation (Jaswal and Akhtar 2019a).

Eventually our commentary was accepted for publication, and approximately a year after the J&A target article was posted online, the final version appeared, followed by thirty-two commentaries and the authors’ responses to the commentaries. Due to the tyranny of the alphabet and my being the first author, our commentary was the thirty-first of thirty-two, appearing on page 48. As we feared, none of the other commentary authors expressed any concern about testimonies derived from FC and RPM. One commentary pointed out that the use of autobiographies by people with autism produced a bias in favor of “cognitively able autistic people, but also, by definition, represent the experiences of people who are motivated to share their stories with others” (Fletcher-Watson and Crompton 2019, 20). But none of the commenters mentioned that the testimonies derived from FC or RPM were not reliable. Indeed, our commentary was the only place in the entire fifty-nine-page document were these discredited techniques were mentioned by name.

J&A responded to all the commentaries at the end of the document, and as might be expected, they defended their use of the suspect quotes. J&A pointed to evidence that individual FC and RPM users had learned to type independently and that FC users in naturalistic observations had said unexpected things or disagreed with their facilitators. In addition, Jaswal reported timing and accuracy data that he had collected on an RPM user that J&A said was so fast (approximately one letter per second) that it was unlikely that he had been cued by someone else. Unfortunately, none of the evidence they cited in defense of FC and RPM were direct tests of authorship, and some was anecdotal. Finally, none of the evidence J&A cited bore directly on the question of whether the quotes they used in their article were valid. Opposed to their view are many simple controlled tests published in peer-reviewed journals showing authorship by the facilitator and not by the person with disability.

J&A’s title for the section replying to us was “Should Testimony from Some Autistic People Be Excluded?” (Jaswal and Akthar 2019b, 57), suggesting that J&A had either missed our point or simply dismissed it. Our entire purpose in writing the commentary was to argue that the voices of people with autism had already been excluded—or, more precisely, had been drowned out by the voices of their facilitators and communication partners.

J&A’s entire response to our commentary can be found in the footnote below.

Adventures in Peer Review

To me, this incident raises a number of questions about the peer review process. Although there are some who would think the J&A article should be retracted, I would have been satisfied if the questionable quotes had been removed—preferably as a result of the peer review process and prior to publication online. Although I do not agree with every aspect of the article, I believe the ideas the authors presented are deserving of publication. I am very glad that we had the opportunity to present our commentary so that the concern was raised, but as the scenario I constructed at the beginning of this article is meant to suggest, I suspect that in many other areas of research the issues we raised would not have been taken so lightly. 

Part of the problem may be the process of choosing peer reviewers. When a target article is submitted to BBS, the authors are asked to provide the names of potential reviewers and commentators. Although this system has been used for years by a number of journals, in this case it may have gone awry. As should be obvious by now, autism is a highly politicized topic. If the authors suggested reviewers who were either supporters of FC and RPM or unfamiliar with the users of the techniques quoted in the article, that might explain the outcome. Whatever happened here, had I, one of my coauthors, or one of the many other scholars who question the use of FC and RPM been a reviewer on this manuscript, the problem might have been spotted much earlier. In which case, our commentary and this article would not have been necessary.

What Responsibility Do Jaswal and Akhtar Have?

With respect to J&A, I come back to two main issues and one smaller one:

1. In my view, J&A should not have used testimony derived from these discredited methods. I want to stress again that this is not just a small controversy. In addition to the policy statements of many professional organizations, FC has been widely covered in the popular press. For example, see recent articles in Slate (“Facilitated Communication Is a Cult That Will Not Die”) and Forbes (“Facilitated Communication Has Been Called an Abuse of Human Rights. Why Is It Still Around?”).

I find it hard to understand why researchers would use such widely discredited evidence in a submission to a prestigious journal. Especially if—as was the case here—it was unnecessary. In their response to our commentary, J&A state, “Our arguments about social motivation in autism would have been the same had we limited ourselves to quoting only speaking people” (Jaswal and Akhtar 2019b, 57), which raises the question of why they used the quotes from people using FC and RPM in the first place.

2. I believe J&A should have been far more open about these matters. I think it was unacceptable to simply use quotes derived from FC and RPM without acknowledging (a) the method by which the quotes were obtained and (b) the controversy surrounding those methods.

3. A smaller item. I believe J&A should not have used the Gralow/jaymee quote given its indeterminate origin, but at very least, they should have identified it more accurately.

Finally, in J&A’s defense, I think they were ill-served by the peer review process. Ideally these issues could have been addressed much earlier.

I find it hard to imagine an area of psychology that is as controversial and politicized as the world of autism. There are very strong feelings on many sides. I acknowledge I probably did not help matters when I used the phrase “Autism Wars” in an earlier column (Vyse 2018). It is clear to me that we all need to work harder at finding a way to talk to each other in a respectful and calm manner. If we can figure out how to do that, I believe that people with autism will be the beneficiaries.

Finally, I hope this case will lead to a discussion of the peer review process. Peer review is one of the hallmarks of modern science. We rely on it to maintain the quality and integrity of our work. It is a very good system, but as the postings on the Retraction Watch website make clear, it does not always produce good results. The challenges faced by peer reviewers are such that the process will probably always be imperfect, but I believe this episode suggests there are ways we could make it a little less imperfect. Perhaps we can all agree that would be a good outcome.

 

---

References

• About Me. N.d. Available online at http://sue-rubin.org/about-me/.
• American Speech-Language-Hearing Association. 2018. Rapid Prompting Method (RPM).  Available online at https://www.asha.org/policy/PS2018-00351/.
• Behavior Analysis Association of Michigan. N.d. BAAM FC Resolutions Compilation. Available online at http://www.baam.emich.edu/baam-fc-resolutions-compilation.html.
• Bigozzi, Lucia, Mirella Zanobini, Christian Tarchi, et al. 2012. Facilitated communication and autistic children: The problem of authorship. Life Span and Disability 15(2): 55–74. (J&A)
• Cardinal, Donald N., and Mary A. Falvey. 2014. The maturing of facilitated communication: A means toward independent communication. Research and Practice for Persons with Severe Disabilities 39: 189–194. (J&A)
• Cardinal, Donald N., Darlene Hanson, and John Wakeham. 1996. Investigation of authorship in facilitated communication. Mental Retardation 34(4): 231. (J&A)
• DiStefano, Charlotte, Wendy Shih, Ann Kaiser, et al. 2016. Communication growth in minimally verbal children with ASD: The importance of interaction. Autism Research 9(10): 1093–1102. (J&A)
• Fletcher-Watson, Sue, and Catherine J. Crompton. 2019. Autistic people may lack social motivation, without being any less human. Behavioral and Brain Sciences 42: 20–21.
• Gernsbacher, Morton Ann, Eve A. Sauer, Heather M. Geye, et al. 2008. Infant and toddler oral‐and manual‐motor skills predict later speech fluency in autism. Journal of Child Psychology and Psychiatry 49(1): 43–50. (J&A)
• Gralow, Christine. 2008. It bears repeating. The New York Times (October 15). Available online at https://lessonplans.blogs.nytimes.com/2008/10/15/it-bears-repeating.
• Higashida, Naoki. 2013. The Reason I Jump: The Inner Voice of a Thirteen-Year-Old Boy with Autism. New York: Random House. (J&A)
• Jaswal, Vikram K., and Nameera Akhtar. 2019a. Being versus appearing socially uninterested: Challenging assumptions about social motivation in autism. Behavioral and Brain Sciences 42: 1–14.
• ———. 2019b. Supporting autistic flourishing. Behavioral and Brain Sciences 42: 51–59.
• Kedar, Ido. 2012. Ido in Autismland: Climbing out of Autism’s Silent Prison. Sharon Kedar. (J&A)
• ———. 2018. I was born unable to speak, and a disputed treatment saved me. Wall Street Journal. Available online at https://www.wsj.com/articles/i-was-born- unable-to-speak-and-a-disputed-treatment-saved-me-1537723821. (J&A)
• Mostert, Mark P. 2001. Facilitated communication since 1995: A review of published studies. Journal of Autism and Developmental Disorders 31(3): 287–313.
• ———. 2010. Facilitated communication and its legitimacy—twenty-first century developments. Exceptionality 18(1): 31–41.
• Tammet, D. 2006. Born on a Blue Day: Inside the Mind of an Extraordinary Autistic Savant. London: Hodder & Stoughton. (J&A)
• Tuzzi, A. 2009. Grammar and lexicon in individuals with autism: A quantitative analysis of a large Italian corpus. Intellectual and Developmental Disabilities 47: 373–85. (J&A)
• United for Communication Choice. 2018. Videos of Advanced Learners. Available online at https://unitedforcommunicationchoice.org/videos-of-advanced-learners/. (J&A)
• Vyse, Stuart. Autism wars: Science strikes back. Skeptical Inquirer 42(6) (November/December 2018). Available online at  /2018/11/autism-wars-science-strikes-back/.

 

---

Note

1.   R3.3. Should testimony from some autistic people be excluded?
   About 30% of autistic people cannot use speech reliably or at all (DiStefano et al. 2016), a disability likely caused, at least in part, by significant difficulties in motor planning, coordination, and/or execution (Gernsbacher et al. 2008a). Some nonspeaking autistic people have learned – after years of instruction and practice – to communicate by typing on a keyboard or pointing to letters on an alphabet board. Typically, these individuals rely on assistance from another person as they spell, someone who provides physical, emotional, and/or attentional support (Cardinal & Falvey 2014).Vyse, Hemsley, Lang, Lilienfeld, Mostert, Schlinger, Shane, Sherry, & Todd (Vyse et al.) objected to our inclusion of testimony from several nonspeaking autistic people who communicate with assistance. In experimental settings, messages produced by people who communicate in this way have been influenced by those assisting them (but see Cardinal et al. 1996). Vyse et al. argued that the testimony of the nonspeaking people we quoted therefore could not have been their own.

   We recognize that nonspeaking individuals who communicate with assistance could unintentionally or deliberately be led to produce testimony that is not their own. Indeed, there is always a possibility that a person’s testimony – spoken, signed, written, or written with assistance—could have been influenced in some way. But there are a number of reasons to believe that testimony produced with assistance, including the testimony used in our target article, can reflect a nonspeaking autistic person’s own thoughts. First, some people who learned to communicate with assistance have developed the ability to communicate independently (United for Communication Choice 2018); the perspectives they communicate independently are consistent with those they communicated with assistance (e.g., Kedar 2012; 2018). Second, nonspeaking people who communicate with assistance routinely convey—in naturalistic, everyday settings—information that the assistant could not have known (e.g., Bigozzi et al. 2012). Third, analyses of text produced by nonspeaking autistic people who communicate with assistance have shown their writing styles to be different from those of the assistants (e.g., Tuzzi 2009).

   Finally, the speed and accuracy with which some nonspeaking autistic people spell make it unlikely those individuals are merely responding to cues from an assistant. For example, one of us (Jaswal) has collected timing and accuracy data from an individual who points to letters on an 8.5- by 11-inch alphabet board held vertically by an assistant. In one 47-letter utterance (“I feel like world is waiting on me not the other way around”), he pointed to a letter, on average, every 1,085 ms (SD = 381 ms) and made no spelling errors. It seems unlikely that in the span of about a second, he could use a subtle cue produced by the assistant to accurately select 1 of 26 possible letter targets, and then repeat this process 46 additional times consecutively without error. Given that about one-third of autistic people are nonspeaking, that some have learned to communicate with assistance, and that some of them have written on the topic of our target article, we believe our inclusion of their testimony was reasonable and appropriate. Interestingly, in most cases, and although this was not intentional, when we quoted a nonspeaking person to make a particular point, we also quoted a speaking person who made a similar point. For example, on eye contact, we quoted Higashida (2013), a nonspeaking autistic person, who wrote that it “feels a bit creepy, so I tend to avoid it” (p. 25) and Tammet (2006), a speaking autistic person, who explained that it feels “strange and uncomfortable” (p. 75). The content of the quotes from nonspeaking autistic people was consistent with the content of the quotes from speaking autistic people. Our arguments about social motivation in autism would have been the same had we limited ourselves to quoting only speaking people.