In my last three blogs, I exposed the myth that the events of September 11, 2001 caused widespread mental and physical health damage to persons who were not present in New York or Washington DC and who did not have direct exposure to the events or have family and friends in harm's way. And, despite claims to the contrary, persons who were riveted to the television watching the events did not experience a virtual trauma.

The claims I critiqued were first published in  on the first anniversary of 9/11 and in a follow-up article in Archives of General Psychiatry a few years later. In my first blog, I showed that the data from Internet web surveys reported in the did not establish any widespread acute posttraumatic stress response. In my second blog, I showed the claims in the JAMA article were inconsistent with data from a number of better designed studies. My third blog showed that claims that these events caused a spike in heart ailments were not backed up by the data in the Archives of General Psychiatry article.

There is much to be learned from the publication and publicizing of these claims:

• Peer review is fallible.
• Even seemingly high-quality journals encourage and thrive on hype.
• Professional organizations shamelessly promote claims that heighten the appearance of the need for their services.

Over time, the Skeptical Sleuth will demonstrate how a confirmatory bias dominates the literature. It is not just a matter of individual authors putting a best foot forward and exaggerating the strength of their findings, but of extensive institutional forces that are invested in promoting a particular view of evidence concerning particular topics even when is not justified.

For now, however, it would be a useful exercise to identify some of the strategies that the authors used to present strong claims about their weak findings in a way that discouraged critical scrutiny. Pointing out what was done in these articles becomes a way of alerting readers to similar strategies in other articles and encourages a healthy skepticism about what they read in the journals and how it is portrayed in the media. On the other hand, what we can identify is also a virtual "how-to" set of instructions on getting strong claims based on weak data published. Follow the directions to get attention if you wish, because audacity and immodest claims frequently succeed.

How the article might have escaped critical scrutiny in the review process and subsequently (you might want to compare the actual article to my analysis below, using this link):

• The introduction that made authoritative statements that aren't backed by data.

Directly relevant data from other sources were not introduced, but we have the following statement which is not evidence-based-

It has been argued that this national trauma "influenced and will continue to influence the clinical presentation of patients seeking health care services" across the country and that it offers "an unfortunate opportunity to find out more about what something like this does to a country as a whole.

Argued, perhaps, but past arguments are not evidence.

• Comparisons with norms and results of other studies were hampered by the authors' altering of the wording or scoring standardized scales, even while  still claiming the validity associated with the original scale.

The use of standardized scales with establish cutoffs should have allowed reviewers and readers to compare the claims of the authors with published data from other sources. But alterations by the authors meant that such comparisons were not possible.

• The authors claimed a high response rate from a nationally representative sample when actually only a small minority of persons approach to participate in the study  were sitl continuing the frequent web-based assessments by the time the 9/11 study was initiated.

The sample was originally recruited for completion of regular Internet surveys on a variety of topics with the incentive being free Internet or coupons. By the time of the assessments after 9/11/2001 were being made, most participants had dropped out, leaving an unrepresentative sample. It takes a very careful read of the JAMA article and related articles in order to figure this out.

• The authors failed  to present basic statistics (means, standard deviations, basic rates of events, and simple correlations) that would allow independent evaluation of the consistency of their claims with simple statistics, but instead relied on multivariate analyses that do not allow independent evaluation.

 Again, reviewers and readers were prevented from deciding for themselves  whether the results of the study made sense in terms of basic statistics. The multivariate analyses are also suspect, but, here too, it takes a careful read by an expert to figure out that these analyses do not capture what should have been available in the simple statistics.

• The authors failed to reconcile their claims with discrepancies in the available published literature.

At the time of the publication of the JAMA article, there was limited data from other sources for comparison, what the authors discouraged checking their data against the other data.

There is a growing body of literature on reactions to the September 11 events.However, several methodological differences between investigations make comparisons of rates of trauma-related symptoms across studies difficult.

Perhaps. But these methodological differences were introduced by the authors of the JAMA paper altering the administration and scoring of standardized measures. Furthermore, by the time their second article appeared in Archives of General Psychiatry, a lot of discrepant evidence had accumulated, but it was simply ignored.