full moon

The full moon, which has no effect on behavior.

The primary way that scientific knowledge is communicated is by the publication of peer-reviewed journal articles. In psychology, a single article in a prestigious journal such as The Journal of Experimental Psychology: Learning, Memory, and Cognition might be the result of many experiments run over the course of three years or more. The reviewers of these articles are very critical, and the whole process ends up publishing, to the best of its ability, only the most well-conducted, interesting, and significant findings. 

But therein lies a problem. Sometimes an article will be submitted that is well-conducted but fails to find statistically significant results. Let's take an example of the effect of having a pet on a person's health. A study that finds a significant correlation between pet ownership and increased health might get published, but a study that fails to find such a relationship is more likely to not be published (and also less likely to be picked up by the popular press). This is one example of publication bias

The end result of lots of individual instances of publication bias is that we can have the scientific literature contain significant results for something that isn't real, and we end up believing in it, simply because all of the studies that failed to find that something were rejected by the journals.

What Does it Mean for a Finding to be "Significant?"

The way statistical significance works is this: a study is deemed "significant" if the results are likely not due to chance. How likely is likely? In psychology, the threshold is usually set at 5% or 1%. So, for example, a paper that finds significant differences between the grades of two different classes is essentially saying that we can be 95% or 99% confident that the difference observed is real and not because of some error in measurement, such as only testing the smart kids in one class and the average kids in the other.

Okay. Let's say that 100 scientists study something that is not real, such as the effect the full moon has on behavior. If we have a threshold of 1%, then by chance one of those studies will come out with a significant result! The scientist who ran that study will be thrilled. She will not know about the other studies being done, and probably never will-- publication bias will make sure those studies are not published. 

People read this article and try to build on it, or replicate it. 1000 more studies are done. This time 100 of them find significant results(by chance, of course, because the moon effect is not real), and might get published. All of a sudden someone can find 101 peer-reviewed, scientific studies that show evidence for something that does not exist

A scientific "fact" was created out of chance and publication bias. 

The File Drawer Effect and Prophetic Discrimination

It's not just the journals and their reviewers that contribute to publication bias. The scholars themselves will often not bother to even submit a write-up of a study that failed to find significant results. This is called the file drawer effect. In some sense, this is prophetic discrimination (Wilner & Jordan, 2009): the scientist discriminates against her own study because she predicts that the reviewers will discriminate against it, so why waste everyone's time? The study goes into the file drawer, never to be submitted. 

I use the pronoun "she" in part because women in science tend to be more critical of their own work, and tend to publish higher quality articles, but with less frequency. As a result, they generally have fewer, but better, publications (Pinker, 2008, p188). People who only want their names on the highest quality work appear to be scholars with integrity, but an unintentional consequence of this is an increased file drawer effect.

Because it didn't get published, lots of other scientists can now waste their time trying the same experiment. How much effort and time has been wasted by scientists all over the world needlessly repeating hopeless experiments simply because they didn't know that it's failed so many times before?

There also might be a deeper motive not to publish the findings. A drug company, for example, has no financial interest in publishing data that fail to show a drug's effectiveness. I have a friend who knew someone who was hired by an animal welfare organization to investigate how packing chickens together affected their welfare. She found no difference, and the organization then tried to prevent her from publishing the results. (By the way, you should always be skeptical of "friend of a friend" stories.)

Tentative Reason For Hope

Part of publication bias has been a result of limited publishing resources. Journals can only be so long. People would only subscribe to a few journals to read. The journals, then, would have an interest in the limited money and attention of their readers. 

But now we have the internet, where the marginal cost of "publishing" more pages or more articles is negligible. Also, people are finding articles more and more through search, rather than reading cover to cover the journals they get in the mail. In this world, there is no problem with a profusion of studies. With hope, journals will start publishing non-significant results, and publication bias might be attenuated. 

Then we will only have to face the file drawer effect.

Pictured: The full moon. It has been well-established by science that there is no effect of the full moon on behavior. http://en.wikipedia.org/wiki/Lunar_effect

References

I recommend the excellent Wikipedia article on Publication Bias: http://en.wikipedia.org/wiki/Publication_bias

Pinker, S. (2008). The Sexual Paradox: Extreme Men, Gifted Women, and the Real Gender Gap. Random House Canada.

 Wilner, S. & Jordan, J. (2009). Discrimination & the female playwright.  Sept/Oct 46--51.

You are reading

The Science of Imagination

How We Can Catch a Ball When We Don't Know Where It Will Be

Our minds have two tracks, when it comes to physical motion

Might We Live in a Simulated World? (A Response to Manzotti)

It depends on what you consider to be really real.