How did wildly inaccurate claims of an American mental health epidemic after 9/11/2001 survive peer review and get into the prestigious JAMA: Journal of the American Medical Association
-- and even get its author sent off to testify before Congress by the American Psychological Association?
The claims were that an epidemic of posttraumatic stress response had occurred across the nation and that it was self-inflicted and aggravated by people watching too much TV coverage of the events of 9/1. It was further claimed that this posttraumatic stress response registered as a measurable increase in heart problems across the nation in the three years that followed 9/11/2001.
How did these ideas get published and spread? The simple answer is that audacity often succeeds, especially with good timing and publicity. The more complex answer will be revealed in this fourth and last installment of the Skeptical Sleuth coverage of the papers making these claims.
In my last three blogs, I exposed the myth that the events of September 11, 2001 caused widespread mental and physical health damage to persons who were not present in New York or Washington DC and who did not have direct exposure to the events or have family and friends in harm's way. And, despite claims to the contrary, persons who were riveted to the television watching the events did not experience a virtual trauma.
The claims I critiqued were first published in on the first anniversary of 9/11 and in a follow-up article in Archives of General Psychiatry a few years later. In my first blog, I showed that the data from Internet web surveys reported in the did not establish any widespread acute posttraumatic stress response. In my second blog, I showed the claims in the JAMA article were inconsistent with data from a number of better designed studies. My third blog showed that claims that these events caused a spike in heart ailments were not backed up by the data in the Archives of General Psychiatry article.
There is much to be learned from the publication and publicizing of these claims:
- Peer review is fallible.
- Even seemingly high-quality journals encourage and thrive on hype.
- Professional organizations shamelessly promote claims that heighten the appearance of the need for their services.
Over time, the Skeptical Sleuth will demonstrate how a confirmatory bias dominates the literature. It is not just a matter of individual authors putting a best foot forward and exaggerating the strength of their findings, but of extensive institutional forces that are invested in promoting a particular view of evidence concerning particular topics even when is not justified.
For now, however, it would be a useful exercise to identify some of the strategies that the authors used to present strong claims about their weak findings in a way that discouraged critical scrutiny. Pointing out what was done in these articles becomes a way of alerting readers to similar strategies in other articles and encourages a healthy skepticism about what they read in the journals and how it is portrayed in the media. On the other hand, what we can identify is also a virtual "how-to" set of instructions on getting strong claims based on weak data published. Follow the directions to get attention if you wish, because audacity and immodest claims frequently succeed.
How the article might have escaped critical scrutiny in the review process and subsequently (you might want to compare the actual article to my analysis below, using this link):
- The introduction that made authoritative statements that aren't backed by data.
Directly relevant data from other sources were not introduced, but we have the following statement which is not evidence-based-
It has been argued that this national trauma "influenced and will continue to influence the clinical presentation of patients seeking health care services" across the country and that it offers "an unfortunate opportunity to find out more about what something like this does to a country as a whole.
Argued, perhaps, but past arguments are not evidence.
- Comparisons with norms and results of other studies were hampered by the authors' altering of the wording or scoring standardized scales, even while still claiming the validity associated with the original scale.
The use of standardized scales with establish cutoffs should have allowed reviewers and readers to compare the claims of the authors with published data from other sources. But alterations by the authors meant that such comparisons were not possible.
- The authors claimed a high response rate from a nationally representative sample when actually only a small minority of persons approach to participate in the study were sitl continuing the frequent web-based assessments by the time the 9/11 study was initiated.
The sample was originally recruited for completion of regular Internet surveys on a variety of topics with the incentive being free Internet or coupons. By the time of the assessments after 9/11/2001 were being made, most participants had dropped out, leaving an unrepresentative sample. It takes a very careful read of the JAMA article and related articles in order to figure this out.
- The authors failed to present basic statistics (means, standard deviations, basic rates of events, and simple correlations) that would allow independent evaluation of the consistency of their claims with simple statistics, but instead relied on multivariate analyses that do not allow independent evaluation.
Again, reviewers and readers were prevented from deciding for themselves whether the results of the study made sense in terms of basic statistics. The multivariate analyses are also suspect, but, here too, it takes a careful read by an expert to figure out that these analyses do not capture what should have been available in the simple statistics.
- The authors failed to reconcile their claims with discrepancies in the available published literature.
At the time of the publication of the JAMA article, there was limited data from other sources for comparison, what the authors discouraged checking their data against the other data.
There is a growing body of literature on reactions to the September 11 events.However, several methodological differences between investigations make comparisons of rates of trauma-related symptoms across studies difficult.
Perhaps. But these methodological differences were introduced by the authors of the JAMA paper altering the administration and scoring of standardized measures. Furthermore, by the time their second article appeared in Archives of General Psychiatry, a lot of discrepant evidence had accumulated, but it was simply ignored.
- The authors waffled, making strong statements about the strength and significance of findings, only to be contradicted in passing in the discussion section. A critic who zeroed in on the strong statements can readily be undercut by the authors referring to the qualifications expressed in the discussion. However if the authors really accepted their qualifications, they wouldn't have made a strong statements in the first place.
Although we were guided by the DSM-IV in the diagnostic criteria we used for both acute and posttraumatic stress symptoms, we did not measure all criteria for ASD and posttraumatic stress disorder (eg, symptom duration)...While we clearly specified that the symptoms be September 11-specific, it is possible that respondents could have attributed preexisting symptoms to the events of September 11. Whereas such posttraumatic stress symptoms appear to mimic the pattern of symptoms identified in the DSM-IV, most of our respondents were not directly exposed to the trauma and, therefore, cannot meet full DSM-IV criteria (or caseness).
So, although we would like to think that alert reviewers would catch these problems, they often do not read carefully. And JAMA was eager to have an article on mental health effects of September 11 to publish on the first anniversary of that day. Journals now regularly track the number of "hits" of articles posted on their website and use these data to attract advertisers. High rates of "hits" mean the ability to charge higher rates for advertising. Journals are big profitable businesses.
How do we explain the American Psychological Association's promotion of this article and its author even sending the author to Washington DC to testify before Congress? Professional organizations often promote unrealistically high estimates of the need for services from their members. One is hard-pressed to find examples of press releases from professional organizations of claims that the prevalence of the clinical phenomena that their members treat are exaggerated. Higher prevalence is good for business, lower estimates are bad.
There was a lot of money at stake for the treatment of mental disorders associated with 9/11/2001. In the first three years afterwards, Project Liberty received $137 million from the federal government, fueled by the earliest reports of the mental health effects of 9/11. Much of the money received from federal, state and local governments as well as private donors was carelessly spent. For instance, a program associated with Scientology received $900,000 in public funding, including $630,000 from the City of New York.
Finally, the people of the United States were hungry to understand how their lives been changed by 9/11, motivating media journalists and scientific journals to give them answers, even if incorrect answers that catered to their worst fears. "This was the week that America changed," Newsweek columnist Fareed Zakeria wrote in September 2001. "In the midst of the jagged emotions of the moment - horror, rage, grief, - we can all sense that the country has crossed a watershed. But we don't quite know what that means."