Psychology Desperately Needs a Massive Influx of Skepticism
Posted Jun 17, 2018
Doubt has a bad rap. "Doubting Thomas" is a dismissive insult originating from, somewhat ironically, a Bible story about an apostle who refused to believe Jesus was resurrected, till he could see it with his own eyes. This is ironic because many modern people "doubt" the literal truth of this story, not the least, some very prominent Christian theologians. Also, everything from the self-esteem movement to pop psychology to the now-largely debunked research on "power posing" tells us to be confident, that confidence produces success. Unabashed advocacy of confidence is pretty much everywhere.
Although there are surely conditions under which confidence is a good thing, in this essay, I am going to argue that the opposite, doubt and skepticism, are also good things, particularly in science, though I am going to focus on psychological science, because that is what i know best.
Skepticism is crucial for healthy science. What is skepticism? It is the embrace of uncertainty and doubt about everything, including science. The popular button and meme “Question Authority” is true but does not go far enough.
Question authorities, your parents, your priest, your professors, media reports, and scientists. Hell, given our propensity for overconfidence and a myriad of biases, you should question your own experiences.
Although this deserves an essay in its own right, skepticism is founded on free speech and academic freedom. Although these are usually discussed as moral, political, and legal principles (note the “and” there – they are not ONLY moral, ONLY political, or ONLY legal), in this essay, I highlight another reason full throated embrace of free speech and academic freedom are important – they are foundational cornerstones for good science. Skepticism is only possible in environments that fully embrace free speech and academic freedom, because it will sometimes manifest as challenges to power and authority; and those with power and authority rarely embrace being challenged. Hostility from those with formal power (gov't officials, college administrators) is often obvious, but power can also be wielded by eminent senior scientists, and even the mob ("mob" here is not a euphemism for Mafia; it is used to refer to large numbers of people acting to threaten and intimidate others and even taking the law into their own hands; see, e.g., French Revolution, the rise of fascism and communism, and, for more recent examples, mobs of both faculty and students threatening those who purvey views they do not like). Speech and academic freedom are foundational for good science because good science thrives on skepticism. Skepticism can involve challenges leveled at Eminent Scientists and their claims, reports of amazing, dramatic findings!, received wisdom, and canonical ideas (as in “the Virgin Birth is part of the canon of Catholicism” and “the Power of the Situation is part of the canon in social psychology.”). But if expression of controversial ideas, including skepticism, is actively suppressed, even if this happens entirely legally, psychological science suffers.
These three analogies, however imperfect, I hope convey why even deep skepticism is crucial to a healthy psychological science.
THREE IMPERFECT ANALOGIES
1. Weeds. Skepticism is crucial for good science in a way that has some similarities to why removing weeds is crucial for a healthy garden. Bad weeds can choke off healthy plants; bad science can choke off good science if, like a runaway weed, sometimes, bad science becomes popular, highly funded, and adopted as a basis for law, social policy, and personal use. It then takes away attention and resources from good science and leads to impressive promises of personal and social change that end up as dead ends, or worse.
Even plants that have some value like scientific claims that have some truth but are wildly oversold can mostly function as weeds, if they go too far. Consider ivy that covers windows and doors; bamboo that takes over a yard; and scientific claims that have some truth but are wildly oversold.
Science has a long history of bad weeds, everything from bad astronomy theories choking off good ones to bad medicine. Social psychology has a disturbingly impressive track record of such “weeds,” including social priming, the power of the situation, stereotype threat, implicit bias, ego depletion, power posing, stereotype inaccuracy, stereotype bias, grit, delay of gratification, the Stanford Prison Experiment, facial feedback, and more (and more and more).
2. Athletic opponents. Serious athletic competition is stressful – that is why everyone from amateurs to professionals sometimes choke in big situations. Michael Jordan has choked and Roger Federer has choked.
I have been a serious amateur tennis player for 35 years, but lord knows, I have choked. In fact, I have a favorite personal choking story. How can choking ever be good?
I was in a mixed doubles tournament, this time, with my adult daughter with disabilities (this was a regular USTA tennis tournament, it was about winning, no one was going to be chivalrous or generous). We got to the finals. We split sets and were in a deciding tiebreaker (first to 10 points would win the set, the match, and the tournament). We got steamrolled and dropped the first 3 points. 0-3. We settle down, grind and get to 3-5. They hit a soft ball in the middle of the court, and, smelling an even match, I rip the ball off my backhand (my best shot). Into the net. Instead of 4-5, it is now 3-6, and we are now staring down the dark tunnel of defeat.
But then we refocused again. 4-6. 5-6. 6-6. 6-7. She holds both her serves! We take the lead 8-7! At 9-7, with a chance to win, I get the exact same soft ball in the middle of the court that I choked on earlier. I think, “Not again, just hit a basic solid, smooth shot.” I do. Our opponent did not return it. Point, game, set, match, and tournament! (my daughter ran off the court into her boyfriend’s arms, like something out of a movie).
Yes, you can have a favorite choking story. Because it is not about choking, but about overcoming choking. About screwing up and improving, getting it right next time.
Which gets us to science, which is also about getting it right. Our athletic opponents bear some similarity to our intellectual “opponents” – be they scientists who disagree with us or even just ideas or theories that “compete” with, or constitute plausible alternatives, to those we believe are true or even have promoted in our own scholarship.
Scientists have a choice: They can either avoid engaging with alternative perspectives, which, like beating up on a weak opponent, probably feels really good in the short term. But one is not likely to actually improve one’s game primarily by beating up on weak opponents. One is almost sure to improve one’s sports performance if one competes against strong opponents. Similarly, in science, avoiding strong alternatives virtually insures that arguments and evidence for their conclusions are weak. In contrast, the term strong inference was coined over 5 decades ago to reflect the idea that engaging alternative views and evidence produces stronger science, points also made by political thinkers (JS Mill), philosophers (Popper), and even some famous sociologists (Merton) and psychologists (Meehl, Vazire).
Social psychological science, however, has a long history of mostly failure (with occasional exceptions) to engage with alternatives. Evidence that self-fulfilling prophecies and stereotype biases are often weak fragile and fleeting is mostly ignored, as is evidence for rationality and accuracy in social perception. Evidence of a serious validity crisis in social psychology is summarily dismissed as overblown despite a mounting body of failed replications and strong alternative explanations to highly influential and canonical claims.
3. Invisible pollution. When obvious, pollution, is often disgusting and its dangers obvious – stinky bus fumes, rivers covered in trash, coastal oil spills. But much pollution is invisible, tasteless, and odorless (e.g., carbon monoxide, CO) – but nonetheless potentially even more dangerous and deadly than the disgusting pollution that is obvious (you can die if you leave your car on in a closed garage; and CO contributes to climate change). But it is precisely because CO is invisible that its dangers are less obvious to the naked eye and often requires sophisticated science to detect (not counting that grim garage business).
In the same way, bad studies are often very difficult to detect. They look a lot like good studies! They are published in peer reviewed scientific journals! They have Scientific Methods (e.g., experiments). They have Statistics! Their authors usually have PhDs – which means they can often write pretty well using sophisticated-sounding polysyllabic technical terms. Their arguments claim the mantle of logic. And they even make compelling arguments for completely bogus or ephemeral phenomena.
It does not matter whether this is done intentionally, the way a chemical manufacturer might dump poison into the local stream because that is the cheapest way to dispose it; or unintentionally, the way someone might die from CO poisoning in their garage; or, even as an unfortunate byproduct of something that is perceived as necessary (e.g., as you drive your car knowing that its emissions pollute because you have no serious alternative to getting around).
The pollution occurs no matter the awareness or intentions of those polluting.
In somewhat the same way, bad studies pollute the scientific record. They are very difficult to distinguish from good studies (though just as methods have been developed to detect pollution, in psychology, there has been a recent flurry of development of methods to detect bad studies). As a result, such scientific pollution taints the entire scientific environment. When even scientists are unable to figure out what is true and what isn’t, how is a layperson supposed to figure it out? And if a significant part of the scientific environment becomes “polluted” by bad studies, why should anyone believe anything that the field produces?
Although there are probably never guarantees, deep skepticism provides an answer to that question. Very little in social psychology – or elsewhere – should be believed until it has been subjected to a mountain of skeptical tests. Science reformers have been developing such tests for some time now, and they include things like:
1. Large sample, registered replication reports – where articles are accepted for publication based on the importance of the topic, the quality of methods, and clear a priori articulation of the statistics to be used to test the hypotheses, but before the data are collected, to insure against both publication biases and cooking the data.
2. Forensic analyses. There are now whole families of related techniques for ferreting out from the statistics in published studies alone how credible they are.
I will end this with a model of how anyone, including nonexperts, can figure out whether psychological science has most likely produced actual facts (this is my variation on a model first developed by Chris Chambers, whose book, The Seven Deadly Sins of Psychology, addresses some of these issues).
If it is not near the top, it might be true, but you have insufficient reasons to believe it is true. And precious little in social psychology is near the top.
Given that pre-registered tests of psychological hypotheses is a relatively new development, this model is probably incomplete – there are findings in social psychology that predate the science reform movement that I consider credible. You can find many of them listed here, in this earlier essay, Are Most Social Psychological Findings False?
If you enjoyed this article, I encourage you too follow me on Twitter, where I address issues like these regularly, and post links to essays, articles, and blogs both by me and by others on these sort of topics.