Why We Believe in Studies, Even After Data Comes Up Short
The answer is unexpected. Literally.
Posted Apr 09, 2018
Frank Sulloway, MacArthur winner and author of Born to Rebel: Birth Order, Family Dynamics, and Creative Lives, enjoyed widespread renown for his theory on the influence of birth order on personality. In short, his hefty, 672-page work explored how first-born children were likeliest to conform to social conventions, while the last-born were far more prone to rebel against them. First-borns, Sulloway argued, scored highly on conscientiousness and were more amenable to their family’s wishes, values, and standards. In contrast, last-borns scored highly on openness to experience, one of the personality traits associated with adventurousness, as well as unconventional and downright rebellious behavior.
Amy Cuddy’s work on power posing not only attracted one of the largest audiences for a TED talk but also snagged the attention of every media outlet from the New York Times to CBS News. By assuming a high-dominance posture, Cuddy’s team argued, anyone could enjoy increased testosterone—the hormone associated with dominance—and decreased cortisol, lowering stress. Cuddy’s initial experiment asked 42 participants to hold a high or low power pose for one minute, then tested participants’ saliva 17 minutes later—neuroendocrine markers that made for more compelling data than mere self-reported feelings of increased self-confidence and decreased stress.
What does Sulloway’s work have in common with Cuddy’s? Both of them create models for understanding behavior that resonate powerfully with readers of all stripes. Yet their research didn't add up.
Replication studies have dismantled the notion of birth order determining personality. Sulloway estimated absolute correlations for openness at .40 and .35 for conscientiousness. However, Damian and Roberts, in a study of 377,000 U.S. high school students, found absolute correlations at .04 for conscientiousness. For the correlation between birth order and personality traits, Damian and Roberts found a lofty .02 absolute correlation. In other words, Sulloway and everyone who’s believed his or her desire to follow or shrug off conventions stems from birth order has been wrong...for over 20 years.
Similarly, Cuddy’s cottage industry, built around power posing, collapsed under replication studies like Ranehill et al.’s work that studiously reproduced every procedure from Cuddy’s earlier study methodology. In contrast to Cuddy’s statistically significant elevations in testosterone levels and decreases in cortisol, Ranehill’s team found no effects whatsoever between high and low power posing and testosterone or cortisol levels.
Yet neither Sulloway’s nor Cuddy’s work received the public drubbing of, say, cold fusion.
Why do we believe unlikely theories, even in the face of data? The answer is surprising—literally. In 1971, Murray Davis identified that interesting theories surprise us by deviating from our expectations, in the same way that surprise or incongruity with existing assumptions generates memorability. In a 2017 study, I also discovered that articles that violated expectations were recalled correctly by participants as long as five days after a single, brief exposure to a short paragraph. In contrast, participants who received the mundane rendering of the same study—with all incongruity removed—were largely unable to identify the meaning of the study after immediate reading, let alone days later.
The upshot? We’d rather believe in a surprising finding than in the less-compelling data that adds up.
For writers with data that overturns common assumptions, runs counter to common sense, or provides surprise—that, for example, standing with your hands on your hips for a minute can raise your testosterone levels and make you more dominant—never bury the surprising discoveries. (Just be prepared for the most surprising findings to become the subject of replication studies.) Instead, foreground them. You’ll make your work likelier to get published and to reach wider audiences.
Carney, D. R., Cuddy, A. J., & Yap, A. J. (2010). Power posing: Brief nonverbal displays affect neuroendocrine levels and risk tolerance. Psychological Science, 21(10): 1363-1368.
Damian, R. I., & Roberts, B. W. (2015). The associations of birth order with personality and intelligence in a representative sample of US high school students. Journal of Research in Personality, 58: 96-105.
Davis, M. S. (1971). That's interesting! Towards a phenomenology of sociology and a sociology of phenomenology. Philosophy of the Social Sciences, 1(2): 309-344.
Douglas Y. (2017) Do paradoxes prompt better attention and recall? Implications for publishing and disseminating academic research. International Journal of Business Administration, 8: 45-54.
Loewenstein G. (1994) The psychology of curiosity: A review and reinterpretation. Psychological Bulletin, 116: 75-98.
Pezzo MV. (2003) Surprise, defence, or making sense: What removes hindsight bias? Memory 11: 421–441.
Ranehill, E., Dreber, A., Johannesson, M., Leiberg, S., Sul, S., & Weber, R. A. (2015). Assessing the robustness of power posing: No effect on hormones and risk tolerance in a large sample of men and women. Psychological Science, 26(5): 653-656.
Sulloway, F.J., 1995. Birth order and evolutionary psychology: A meta-analytic overview. Psychological Inquiry, 6(1): 75-80.
Sulloway, F.J., 1996. Born to rebel: Birth order, family dynamics, and creative lives. Pantheon Books.