Do Spectacular Findings Lead to Spectacular Frauds?

Invisible gorillas, feminist bank tellers, and the real problem with psychology.

Posted Oct 11, 2013

In 1994, the NYPD introduced a system to track crime data called Compstat. Although it's hard to pinpoint exactly why, crime dropped like an anvil for many consecutive years. Compstat seems to have been a huge success.

At some point everyone started expecting crime rates to drop every year. But then crime rates stopped dropping. The rates were still low, but that wasn't good enough anymore. There was an incentive for police captains to manipulate their data in order to keep dropping. Some captains resisted this pressure—probably most—but some took steps to artificially boost their numbers (This American Life presents a dramatic case of this problem). 

In short, when amazing, crazy data become the norm, everything else seems tiresome and dull. And that can be a big problem.

Look at Las Vegas hotel bathrooms. Every time one hotel comes up with a more opulent, ridiculous bathroom, another hotel has to top it. It's a endless cycle of one-upmanship. Just like publicly traded companies, which have to do better than last year—every year—or suffer the wrath of investors.

Now look at psychology. If you peruse an intro psych textbook, or read about psych in the popular press, or, heck, even read the TOC of some high-profile academic journals, it's full of jaw-dropping findings like these:

  • Feminist bank tellers
  • Invisible gorillas
  • Castle/tumor analogy problem
  • Law of small numbers
  • Stanford prison study
  • Milgram conformity studies
  • Marshmallows predict virtually everything
  • Loftus memory manipulation studies
  • Deese-Roediger-McDermott
  • Stereotype threat
  • Priming elderly makes people walk slowly
  • Etc.

To me, each of these findings is simply amazing. (Assuming it's true—which is a safer assumption in some cases than others.)

Invisible gorilla, for example. It's not just that people sometimes fail to notice things. It's that lots of people fail to notice something that should be blindingly, mind-numbingly obvious even to a one-eyed giraffe. Or take the fact that people think it's more likely that Jane is a feminist bank teller than that she's a bank teller. It's not like a few people are a little bit wrong. Almost everyone believes something that, if you think about it for a few seconds, is obviously and completely impossible. 

Like the amazing data compstat generated in New York, maybe the amazingness of these findings is part of the problem.

And there is a problem. There are recent high-profile cases of psychologists falsifying data. There also seems to be an endless parade of almost unbelievable findings. Many psychologists are worried that some of these are too good to be true—they're fluke findings that aren't replicable.

Maybe it's the compstat/Vegas bathroom problem all over again. Psycholgists are competing for attention in an ever-escalating sea of crazy findings (including the list above).

You don't bring a knife to a gunfight. When merely interesting data aren't enough, researchers have an incentive to "enhance" their findings to make them look spectacular. 

The fact that it's so hard to get a job also doesn't help. As Brian Nosek points out in this article, "The pressure [to get positive results] is very high, very early." And the gap keeps growing between the low number of faculty jobs and the high number of eligible PhDs (see chart below). 

A good scientist is more interested in the truth than making things spectacular (although of course, a spectacular truth is hard to beat). Still, maybe, like Homer Simpson at a garage sale (hmmm.. it only transports matter?), we need to lower out expectations a little bit. 

Follow me on Twitter.

More Posts