Verified by 
In science, as in life, if it seems too good to be true, it probably is.
This is just a brief post to draw attention to the excellent recent article "False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant" by Simmons, Nelson, and Simonsohn. http://pss.sagepub.com/content/22/11/1359
They show that some fairly widespread scientific practices-such as determining sample size on the fly, and assessing the effect of the independent variable being manipulated in an experiment on multiple dependent variables being measured simultaneously-can actually lead to false positive rates as high as 61%! That is to say, if one follows some of the practices they describe, one risks "discovering" something false 61% of the time.
They illustrate the possibility with a study that shows a significant decrease in the chronological age of participants after listening to The Beatles' "When I'm Sixty-Four". That's funny, to be sure-but it also points to a serious problem. I urge every scientist, critical consumer of science, and (perhaps especially) science journalist, to read and reflect on the article. Their diagnosis seems sound, and their suggestions for how to address the issue are reasonable. I'll not repeat them here.
However, I do want to emphasize one thing the authors do not: the risk they report underscores the vital need for replication. We tend to assume that replication is the order of the day in science, but things are not so simple. In some sciences, notably physics, where theories predict specific values for certain measurements, the incentive to replicate is built in. Since later experiments can often improve the precision of measurement, replication is common and represents a clear form of progress. In other sciences, notably psychology, theories instead predict quantitative patterns of results: this value should be larger in this condition than in that one. Here it can seem as though replication provides no new information, and thus represents no real progress. Reflecting this assessment, it is difficult (if not impossible) in psychology and cognitive neuroimaging to get funding for, or to publish, replications. And where replications fail, one is left with the classic conundrum of the null result: did the experiment produce no result because there is something wrong with how one performed the experiment, or because the null hypothesis is in fact true?
It is time to admit across all the sciences that replication is valuable, and null results are informative.
Update 1/21/2012:
There now exists a website dedicated to tracking replication attempts in Psychology: Psych File Drawer. Please use it!
(Thanks to Tal Yarkoni's blog for alerting me to this resource)
Related work of interest:
The Truth Wears Off: Is there something wrong with the scientific method?
http://www.newyorker.com/reporting/2010/12/13/101213fa_fact_lehrer
