Psychology and the War on Referees
Applying lessons from Michael Lewis' podcast to psychology.
Posted Feb 01, 2020
In an early episode of Michael Lewis’ podcast Against the Rules, Lewis describes how NBA stars complain so vehemently against refs these days. The problem isn’t that refs are bad. It’s that they’re too good. The NBA has implemented stricter training regimens for refs, including reviewing all their calls and giving them detailed feedback from the video booth. New technology is being used to provide (i) more objective measures of performance, and (ii) better feedback. As a result, refs have stopped playing favorites with the NBA stars. It turns out that, for years, the best basketball players also got the most favorable treatment from the refs. Now the refs are being evaluated according to a more objective metric, and they’re no longer giving preferential treatment—leading to huge complaints from the people who have to play by fairer rules.
Michael Lewis, who wrote the books Moneyball, The Blindside, and The Big Short, is one of my favorite popular nonfiction writers. Last year, he released the podcast Against the Rules about attacks on the referees in American life. Lewis’ thesis is that fairness is eroding in American society as people are increasingly getting angry at referees. Wherever Lewis turns his eye, he sees this phenomenon: in sports, in government, and in financial institutions.
Ironically, Lewis has the social psychologist Dacher Keltner on his show to discuss this phenomenon—without addressing the fact that anger at the refs is rampant in Keltner’s own field! Social psychology has been the epicenter of the Reproducibility Crisis, which was sparked by a failure on the part of traditional referees: the publication of a set of research studies showing evidence for “psychic powers” (but, in one study, only for being able to predict whether pornographic images were about to be displayed, and only for men) by Daryl Bem. The publication of this study, which showed robust support for borderline absurd claims (claims that flew in the face of hundreds of years of physics and prior science), in social psychology’s then top journal, was a wake up call to many researchers. How could our most prestigious—what we considered our best—journal publish this?
The editors who published Bem’s articles defended their decision on the basis that he met the standards of evidence typical for the journal. But if Bem could find support for the idea that men have special psychic powers for figuring out if pornographic images are about to be displayed, how good were those standards? How good were our referees? Were they letting star researchers like Bem get away with breaking the rules of science?
Bem’s infamous study led other researchers to reverse engineer his formula, demonstrating how commonly used research design and statistical techniques can lead to impossible results. Simmons, Nelson, and Simonsohn demonstrated in 2011 that a combination of four common, seemingly “inoffensive” techniques—like collecting a few extra participants if a study hasn’t shown a significant result, or deciding whether to adjust an analysis for gender differences—can easily lead to an incorrect decision. Researchers were reporting that there was only a 5% chance the effect they reported wasn’t real (e.g., a 5% chance they were just observing a fluke). In reality, it was over 60%. When people used the standards being enforced by the refs—the scientific review process—they were more likely than not to present flukes as real results.
This and other results led to the birth of the Credibility Revolution, which in many ways is a movement towards stronger refereeing. The referees haven’t been doing their job, and we’re left with a league in which all superstars have come to expect that they won’t be called on breaking the rules. Many of the most famous and successful social psychology researchers sound just like the aggrieved, entitled athletes Lewis describes. They are mad that the referees are getting better.
Viewing the current Credibility Revolution as a movement towards restoring strong, legitimate referees in science helps explain the dynamics surrounding the movement. The emphasis on finding mistakes and pointing them out is a way of asserting that the referees are on duty, and scientists need to make sure they’re careful and play by the rules. The large scale replication projects are audits of the current state of the scientific field, which gives more objective feedback about how things are going.
The way we think about people who are criticized by members of the Credibility Revolution can also be understood through the referee metaphor. Like a player who stomps around angrily after getting a call against him, a scientist who complains and personally attacks the critic is seen as a poor sport—and childish. Like a player who grimaces but then accepts the call, a scientist who responds to criticism by being more transparent and acknowledging limitations or errors is seen as behavior maturely and honorably.
The referee metaphor also highlights some of the limitations of the current emphases of the Credibility Revolution. Having good referees doesn’t ensure that great basketball is played; it just ensures that basketball players can’t win by bending or breaking the rules. Similarly, having good scientific referees doesn’t ensure that great science gets done. It just ensures that we don’t gives prizes to scientists for unreliable, incorrect work. The Credibility Revolution is also criticized for being seen as destructive, as opposed to constructive. That’s in the nature of a referee. Referees enforce rules; they don’t come up with new plays.
Michael Lewis’ podcast is worth listening to, because it tells interesting stories about a pervasive problem in U.S. society. People love to talk about how systems and institutions are broken, but then they turn around and criticize the very people who are supposed to make sure those institutions work: the refs. There is no easy solution to the problem offered by Lewis. But a broad-based reform movement driven by early and mid-career people, like the Credibility Revolution, is probably high on the list of non-easy solutions. Let’s hope Lewis takes some time to think about this phenomenon happening right in front of him in the next season, and the solutions psychology is pioneering.