Everybody Is Stupid Except You

The truth about learning and memory.

How to Tell if a Finding Should be a Target for Replication

Which findings should be targets for replication?

It's important to replicate scientific findings--but which ones? You can't replicate them all. Researchers should collaboratively decide on a set of principles we can all follow as we select studies to replicate. Below, I provide a set of principles as a starting point. 

I’ll start with some background from the past week. But, if that’s boring, skip it. 

Background

There is a movement to encourage replication in psychology. Although replication has a long history, an identifiable movement has coalesced recently. Being young, it is naturally still maturing. Hopefully it’ll grow into a robust adult in the coming years, but there’s also a danger it’ll suffer the fate of so many movements (e.g., the cyclical “new statistics” movements over the years): it won’t be adopted on a large scale and the status quo will not change.

Simone Schnall, (University of Cambridge) recently published a measured criticism of the replication movement in it’s current form. This is not to say I agree with everything she said, but she did the research community a favor by having the courage to publish it. Whether you agree or not, it is exactly the kind of conversation we need to be having.

Find a Therapist

Search for a mental health professional near you.

Her post has garnered a lot of attention (after being written about in Science) mainly because she accused replicators of bullying on blogs and social media, a controversial claim. Daniel Kahneman wrote a commentary suggesting rules for replication etiquette. But I’m here to talk about something else. 

Schnall says, in her blog post, that her kind of research—social psychology research on embodied cognition—is unfairly targeted for two reasons: it’s cheap/easy to replicate and the results are surprising. My question is: which findings should we be trying to replicate? I don’t know the answer and this issue is far too important to be answered by me in a blog post. And there’s no recipe. So I welcome further thoughts, etc. But here's a start.

Reasons a finding should be replicated by an independent lab

  1. It has had a big impact. This includes, it’s appeared a lot in scientific literature; it’s in textbooks (and thus becomes common knowledge for everyone in psych); it’s been covered in the media.
  2. It hasn’t been replicated before.
  3. It is old. Why? Because failed replications of old studies are more likely to be sitting in file drawers (than failed replications of new studies) because there has been more time for people to try and fail to replicate.
  4. The original research suffered from clearly identifiable problems. For example, small n; questionable research practices (e.g., data fishing expeditions); etc (there are lots).
  5. The researcher doing the replication is an expert in the field.
  6. The finding is surprising.
  7. Doing a replication is cheap/easy.
  8. You just don’t believe the finding.

I feel fairly strongly about 1-4. Number 5 would be a plus, (at least if someone is replicating me, I would feel this way) but if a given field isn't stepping up, people outside the field are going to have to.

I'm not so sure 6-8 should be given as much weight. If they are, as a field, are we going to end up trusting findings that seem right and/or would take more effort to replicate? And is that a bad thing?

Based on these criteria, it's easy to find tons of findings in cognitive psychology (my field) that should be replicated. Many of these are findings that I have no reason to doubt are valid and replicable. I'm sure the same is true for every field. 

Did I miss something? Am I wrong? Good. Let's figure it out. 

Follow me on Twitter. 

 

Nate Kornell, Ph.D., is an Assistant Professor of Psychology at Williams College.

more...

Subscribe to Everybody Is Stupid Except You

Current Issue

Love & Lust

Who says marriage is where desire goes to die?