It's important to replicate scientific findings--but which ones? You can't replicate them all. Researchers should collaboratively decide on a set of principles we can all follow as we select studies to replicate. Below, I provide a set of principles as a starting point.
I’ll start with some background from the past week. But, if that’s boring, skip it.
There is a movement to encourage replication in psychology. Although replication has a long history, an identifiable movement has coalesced recently. Being young, it is naturally still maturing. Hopefully it’ll grow into a robust adult in the coming years, but there’s also a danger it’ll suffer the fate of so many movements (e.g., the cyclical “new statistics” movements over the years): it won’t be adopted on a large scale and the status quo will not change.
Simone Schnall, (University of Cambridge) recently published a measured criticism of the replication movement in it’s current form. This is not to say I agree with everything she said, but she did the research community a favor by having the courage to publish it. Whether you agree or not, it is exactly the kind of conversation we need to be having.
Her post has garnered a lot of attention (after being written about in Science) mainly because she accused replicators of bullying on blogs and social media, a controversial claim. Daniel Kahneman wrote a commentary suggesting rules for replication etiquette. But I’m here to talk about something else.
Schnall says, in her blog post, that her kind of research—social psychology research on embodied cognition—is unfairly targeted for two reasons: it’s cheap/easy to replicate and the results are surprising. My question is: which findings should we be trying to replicate? I don’t know the answer and this issue is far too important to be answered by me in a blog post. And there’s no recipe. So I welcome further thoughts, etc. But here's a start.
I feel fairly strongly about 1-4. Number 5 would be a plus, (at least if someone is replicating me, I would feel this way) but if a given field isn't stepping up, people outside the field are going to have to.
I'm not so sure 6-8 should be given as much weight. If they are, as a field, are we going to end up trusting findings that seem right and/or would take more effort to replicate? And is that a bad thing?
Based on these criteria, it's easy to find tons of findings in cognitive psychology (my field) that should be replicated. Many of these are findings that I have no reason to doubt are valid and replicable. I'm sure the same is true for every field.
Did I miss something? Am I wrong? Good. Let's figure it out.