Verified by 
It would be great to believe in unicorns. And that picture to the left looks real, doesn't it? Of course, we know that belief in animals such as unicorns and dragons is completely silly, right?
Well, not really. Often, there were good bases for the origins of such myths. The rhinoceros, oryx, and several extinct mammals could have served as bases for what became myths about unicorns. Dinosaur skeletons found in China inspired belief in dragons. In many cases, there once really was plausible evidence reasonably interpretable as consistent with the myth, and something about the myth itself that captured people's imagination. Widespread belief in unicorns in scholarly circles lasted well into the 19th century.
Do scientific myths arise in a similar way? A study finds evidence for some sort of dramatic phenomena that captures people's attention. People then run with it, exaggerating it all out of proportion. Sometimes, the phenomenon is ultimately found not to exist at all; to have been a phantom, an illusion. The proponents of scientific myths rarely lie or intentionally mislead; they generally believe in the myths they propagate (as did the medieval scholars describing unicorns). I suspect that a combination of a sincere belief in a myth, genuine scientific expertise, and an engaging writing style will be found very convincing by many people, even scientists.
Here are two classic studies from social psychology that subsequent research has found very difficult, or impossible, to replicate:
1. Stereotypes act as hypotheses that lead to their own confirmation? This was the conclusion reached by Darley & Gross (1983) when they found that people did not apply social class stereotypes when judging a 4th grade girl unless they also viewed her taking a math test. In that case, social class stereotypes biased judgments: people judged the girl as more motivated and intelligent when they believed she was from a higher SES background.
Two attempts at exact replication have failed (Baron, Albright, & Malloy, 1995). Indeed, they found the exact opposite pattern: 1. Stereotypes only influenced people's judgments when they had no other information about her; 2. When they viewed her math performance, their stereotypes had no effects at all. Furthermore, this latter pattern is more consistent with findings from hundreds of studies and several meta-analyses than is the Darley & Gross study's results (Jussim, 2012). Although the Baron et al (1995) study has been cited fewer than 30 times, the Darley & Gross paper has been cited over 600 times since 1996.
2. The famous Bargh et al (1996) study that found that priming people with the elderly stereotype led them to walk more slowly has been cited over 2000 times. A recent dramatic failure to replicate (Doyen et al, 2012) created quite a splash (links at end of this blog). Furthermore, once that study came out, lots of other people reported their own, unpublished, failures to replicate.
In his defense, on his blog Bargh cited two published replications (Cesario et al, 2006; Hull et al, 2002). Neither, however, found statistically significant main effects for elderly primes (though both were borderline, with p-levels ranging from .06 to .13 and the means in the predicted directions; both also found statistically significant evidence of the predicted priming effect among subgroups of participants). If this constitutes "replication" it strikes me as very weak indeed.
Why is all this important? One of Bargh's responses to the Doyen failure highlights exactly why this is important. From Bargh's blog: "Doyen et al.'s replication of our 1996 study had many important differences from our procedure, all of which worked to eliminate the effect."
The big issues involve the common social psychological emphasis on the power of stereotypes to bias judgment and Bargh's claims about the "unbearable automaticity of being"—about human thought, motivation, feelings, and behavior being overwhelmingly dominated by unconscious processes. No one doubts that stereotypes can lead to biases or that "priming occurs." Nonetheless, why is it so hard to reproduce both of these particular effects? Why is it so easy to "stumble" onto limitations to the effects? A common and powerful social phenomena should be very easy for almost anyone to obtain (as is stereotype accuracy or the power of individuating information over stereotypes, but I digress...).
Are stereotypes leading to their own confirmation, and elderly primes leading people to walk slowly modern, social psychological examples of unicorns? In both cases, there is something there, just less than often claimed; just as if you take away the magic and horn, there is a very real horse under that unicorn. Philip K. Dick once wrote: "Reality is that which, when you stop believing in it, doesn't go away." What shall we call those things that do go away?
References
Bargh, J.A., & Chartrand, T. L. (1999). The unbearable automaticity of being. American Psychologist, 54, 462-479.
Bargh, J.A., Chen, M., & Burrows, L. (1996). Automaticity of social behavior: Direct effects of trait construct and stereotype activation on action. Journal of Personality and Social Psychology, 71, 230-244.
Baron, R. M., Albright, L., & Malloy, T. E. (1995). The effects of behavioral and social class information on social judgment. Personality and Social Psychology Bulletin, 21, 308-315.
Darley, J. M., & Gross, P. H. (1983). A hypothesis-confirming bias in labeling effects. Journal of Personality and Social Psychology, 44, 20-33.
Doyen, S., Klein, O., Pichon, C., Cleeremans, A. (2012). Behavioral priming: It's all in the mind, but whose mind? PLoS One, 7(1): e29081. doi:10.1371/journal.pone.0029081, retrievable at: http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0029081
Jussim, L. (2012). Social perception and social reality: Why accuracy dominates bias and self-fulfilling prophecy. NY: Oxford University Press. (Abstracts, first chapter, and excerpts available at: http://www.rci.rutgers.edu/~jussim/)
Unicorn. Retrieved on 7/10/12 from: http://en.wikipedia.org/wiki/Unicorn.
Bargh blogs defending his findings and critiquing the Doyen et al study can be found at:
http://www.psychologytoday.com/blog/the-natural-unconscious/201203/nothing-in-their-heads
http://www.psychologytoday.com/blog/the-natural-unconscious/201203/angry-birds
Some additional reports of unpublished failures to replicate the Bargh et al (1996) walking slowly down the hall study:
http://psychfiledrawer.org/chart.php?target_article=1&type=failure
https://groups.google.com/forum/#!topic/openscienceframework/2XhZ6s9otQg
Some good critical analyses of the whole situation can be found at: https://plus.google.com/107191542129310486499/posts/VJH8wXxxc3f
http://hardsci.wordpress.com/2012/03/12/some-reflections-on-the-bargh-doyen-elderly-walking-
priming-brouhaha/ http://blogs.discovermagazine.com/notrocketscience/2012/03/10/failed-replication-bargh-psychology-study-doyen/
