Are Journal Editors Responsible for Poor Quality COVID-19 Research?
Part seven of my series on the poor quality of COVID-19 mental health research.
Posted Feb 21, 2021
Key Points: Science journals have rushed to publish research during the pandemic. But a good percentage of the papers reveal shaky analysis, unsupported claims, meaningless insight, and even embarrassing typos. Poor research greatly harms the public, especially at times of crisis.
The majority of mental health research related to the COVID-19 pandemic has been poor-quality. As the editor-in-chief of the Journal of Anxiety Disorders wrote, it’s often been “Garbage in, garbage out” (Asmundson & Taylor, 2021). It wasn’t just the low-quality journals either. Researchers who conducted casual, non-rigorous studies are of course largely responsible for this, but there is perhaps a bigger problem.
The bigger problem may be the editors who published the garbage. Researchers submit garbage to journals all the time. The worst of it usually gets weeded out by peer-review. The peer-review system has its flaws, but it works well most of the time. During the COVID-19 pandemic, the normal peer review was changed at many journals that were trying to publish more rapidly.
The normal peer-review process was changed
The editor-in-chief of the Asian Journal of Psychiatry described how he changed the review process for the pandemic. After inviting submissions, he received over 550 in six weeks and published 52 of them. The changes to the normal peer-review process needed to accomplish that speed were astounding. The initial reviewers provided “cursory feedback” within two days. The authors were given one week to make revisions. Re-review after revisions had been minimized (Tandon, 2020).
The literature was flooded with studies that were meaningless and misleading. My series of blog posts showed this, and I’m not alone in saying that. Hundreds of studies on mental health were published in a short span and most were of such poor quality that they weren’t worth summarizing (DEPRESSD Project). One study using three-item or nine-item measures disseminated carelessly by social media apps was one too many.
Case reports have suffered too
Case reports were of baffling quality. The fictional disorder of “coronophobia” was diagnosed in at least two case reports. A 23-year-old female in Indonesia came to the attention of doctors (it’s not stated how) with extreme anxiety attributed to the appearance of COVID-19 in her country. However, there was no report of whether she suffered anxiety prior to the pandemic (Wulandari et al., 2020). A 38-year-old female in Peru became suddenly psychotic after a visit to a dentist in March 2020. She feared she had been infected because the dentist didn’t wear a mask and had recently traveled to France (Huarcaya-Victoria et al., 2020).
An 18-year-old Chinese male visited Wuhan in late 2019 and then two days later developed physical symptoms consistent with COVID-19. His negative COVID tests were puzzling until his doctors discovered he had a six-year history of depression. Then they concluded his fear of being infected triggered all the other symptoms including fever, sweating, and cough (Fu & Zhang, 2020). We’ve known for a long time that stress can manifest as psychosomatic symptoms. Life is full of stressors; it is not evident that COVID-19 stress is a unique variety of stress. Why the editor felt this was a new contribution to psychiatry is puzzling.
Non-mental-health professionals were suddenly mental-health experts
Some of the commentaries in mental health journals that screamed the loudest about our being in the midst of a psychological pandemic were authored by non-mental health professionals. A dentistry school biostatistician suggested that psychological “disorders are more prevalent in non-members of medical teams than in frontline healthcare professionals” through vicarious trauma (Ghaffari & Mortezapour, 2020). A health policy researcher predicted that the pandemic will have “profound mental health impacts that pervade racial, ethnic, and class lines in the United States“ (Purtle, 2020). A professor from the School of Management in Malaysia claimed economic hardships for immigrants contributed to excessive mental health problems, yet cited no mental health research (Mia & Griffiths, 2020). Why would editors publish such extraordinary, unsupported claims from individuals with no mental health expertise?
Proof-reading seemed non-existent
In many cases, it appeared that absolutely no proofreading was done by editorial assistants. Proofreading is important, especially when so many studies were from different countries. In a previous blog post (11/6/2020), I noted this sentence in a study published in the journal PLOS ONE, “Huge citizens expose to social media during a novel coronavirus disease (COVID-19) outbroke in Wuhan, China.” (Gao et al., 2020). I did not know that the mental health of huge citizens in China was impacted differently than small citizens. And I learned that outbroke is a valid Scrabble word, but not in this context.
There are too many of these grammar errors to summarize, but here’s another example. "This paper suggests the applying HBM to COVID-19 in mitigating behaviors which provokes anxiety and fear and converts individual beliefs informed by preconceived impressions of a perceived threat and direct cues of perceived benefits from perceived barriers to action inform behaviors (through perceived self-efficacy)” (Mukhtar, 2020). Got that?
Retractions are everywhere
The problem of research quality during the pandemic was not limited to mental health. As of February 13, 2021, there have already been nearly 70 journal publications on medical topics retracted by journals because either fraud, sloppy mistakes, or unexplainable statements were discovered post-publication and pre-publication peer review failed to uncover them (Retraction Watch 2/13/21).
If you’re a researcher in any specialty, and you’re not following Retraction Watch, you’re missing one of the great stories in the history of science right now. When the good folks at Retraction Watch started tracking retractions in 2010, retractions were thought to be rare. Not so. Science is being corrupted daily by paper mills, predatory journals, fake authorship, and fake data. These are not conspiracy theory hyperbolic claims. These are based on hundreds of retractions per year by the journals themselves. And those are just the ones that got caught. Much of the current concern stems from Chinese universities that often use a pay-for-paper model with their faculty, but the problems occur at some level everywhere.
For example, the Asian Journal of Psychiatry retracted one of those rapidly published papers because the Chinese authors had managed to get it published in three separate journals. The duplicates were discovered when a staffer at a different publisher took the time to run a plagiarism check (Retraction Watch 10/29/20).
The harm of dodgy science
You have to wonder about the politics. The responses to the pandemic in every country were totems of political ideology. As psychologist Philip Tetlock has noted, scientists are politicized in ways they are often dimly aware of (Tetlock and Mitchell, 2015). During COVID-19, objectivity was abandoned in many instances.
This last year raises questions about the resilience of science journals to withstand political times and this age of weaponized ideologies. The behavior of many journal editors seems to indicate, “Yes, we stand for truth through the rigors of the scientific process, just not all the time.” Gorman and Gorman were calling for journal editors to be more responsible well before the pandemic. “Scientific journal editors must always bear in mind the possibility that articles they accept for publication will emerge on the Internet and be seen – and misinterpreted – by nonscientists who are not the intended audience” (Gorman and Gorman, 2017, p 259).
Was this the test run for changing the peer-review process?
The peer-review system through all specialties has been under attack from the explosion of pre-print repositories, lack of good framing metrics (Fleerackers et al., 2021), open access models, predatory journals, greedy reviewers demanding to get paid, and paper mills used by researchers in China under a pay-per-paper model (Else, 2020). Many reformist scientists have called for radical changes to the peer-review process like the type we’ve seen during COVID-19.
We might think of this COVID-19 period as a test run for what happens when peer review is relaxed. The result: nothing good. Neither truth nor treatments were discovered more accurately or more rapidly.
When it comes to science, speed kills. It kills integrity and trust.
Asmundson GJG, Taylor S. Garbage in, garbage out: The tenuous state of research on PTSD in the context of the COVID-19 pandemic and infodemic. Journal of Anxiety Disorder 78, March doi.org/10.1016/j.janxdis.2021.102368
DEPRESSD Project. https://www.depressd.ca/research-question-1-symptom-changes (accessed 1/31/2021).
Else H (12/16/20). How a torrent of COVID science changed researchpublishing — in seven charts. Nature 588, 553 (2020) doi: https://doi.org/10.1038/d41586-020-03564-y
Fleerackers A, Riedlinger M, Moorhead L, Ahmed R, Alperin JP (2021): Communicating Scientific Uncertainty in an Age of COVID-19: An Investigation into the Use of Preprints by Digital Media Outlets, Health Communication, doi: 10.1080/10410236.2020.1864892
Fu R, Zhang Y (2020). Case report of a patient with suspected COVID-19 with depression and fever in an epidemic stress environment. General Psychiatry 2020;33:e100218. doi:10.1136/gpsych-2020-100218
Ghaffari M, Mortezapour A. (April 7, 2020). Letter to Editor: Vicarious traumatization in the general public, members, and non-members of medical teams aiding in COVID-19 control. Brain, Behavior, and Immunity 87 (2020) 25–26, https://doi.org/10.1016/j.bbi.2020.04.006
Gao J, Zheng P, Jia Y, Chen H, Mao Y, Chen S, et al. (2020), Mental health problems and social media exposure during COVID-19 outbreak. PLoS ONE 15(4): e0231924. https://doi.org/10.1371/journal.pone.0231924
Gorman SE, Gorman JM (2017). Denying to the Grave: Why We Ignore the Facts That Will Save Us. Oxford University Press: New York NY.
Huarcaya-Victoria J, Herrera D, Castillo C. Letter to the Editor. Psychosis in a patient with anxiety related to COVID-19: A case report. Psychiatry Research Psychiatry Research 289 (2020) 113052, https://doi.org/10.1016/j.psychres.2020.113052
Lozada-Martínez I, Bolaño-Romero M, Moscote-Salazar L, Torres-Llinas D (2020). Letter to the Editor Regarding “Two Perspectives on the COVID-19 Pandemic Nobody Is Talking About—and It’s Costing Lives” and “Emotional Health in the Midst of the Coronavirus Disease 2019 (COVID-19) Pandemic. World Neurosurgery Volume 144, December 2020, Page 299, https://doi.org/10.1016/j.wneu.2020.08.030.
Mia MA, Griffiths MD (2020). Letter to the Editor: The economic and mental health costs of COVID-19 to immigrants. Journal of Psychiatric Research Volume 128, September 2020, Pages 23-24, https://doi.org/10.1016/j.jpsychires.2020.06.003
Mukhtar S (2020). Mental health and emotional impact of COVID-19: Applying Health Belief Model for medical staff to general public of Pakistan. Brain, Behavior, and Immunity 87 (2020) 28–29, doi: 10.1016/j.bbi.2020.04.012
Purtle J (June 17, 2020). COVID‑19 and mental health equity in the United States. Social Psychiatry and Psychiatric Epidemiology, ahead of print 6/7/20, pp 1-3. https://doi.org/10.1007/s00127-020-01896-8
Rajkumar RP (April 10, 2020). COVID-19 and mental health: A review of the existing literature. Asian Journal of Psychiatry 52 (2020) 102066, https://doi.org/10.1016/j.ajp.2020.102066
Retraction Watch (2/13/21), https://retractionwatch.com/retracted-coronavirus-covid-19-papers/
Retraction Watch (10/29/20), https://retractionwatch.com/2020/10/29/researchers-publish-the-same-covid-19-paper-three-times/
Tandon R (2020). COVID-19 and mental health: Preserving humanity, maintaining sanity, and promoting health. Asian Journal of Psychiatry 51, doi: 10.1016/j.ajp.2020.102256
Wulandari P, Hidayat R. General Anxiety Disorder-Related Coronavirus Disease-19 Outbreak in Indonesia: A Case Report. Open Access Macedonian Journal of Medical Sciences. 2020 Jun 10; 8(T1):36-38. https://doi.org/10.3889/oamjms.2020.4762
Tetlock, P.E. & Mitchell, G. (2015). Why so few conservatives and should we care? Society 52(1) doi: 10.1007/s12115-014-9850-6