What’s the Best-Replicated Finding in Social Psychology?

These days, social psychology is beset by feelings of doom and gloom about its replication crisis. But instead of looking at the dark side, let’s look at some good news. In the spirit of positive psychology, let’s ask the question: What is social psychology’s best-replicated finding?

Any scientist starts by asking about methods. What criteria should be used to pick the best-replicated finding? I will suggest the following. Obviously, if you pick different criteria, you might come to a different winner.

I would think a well-replicated finding would be marked by plenty of significant published findings in support of it, and presumably few or none in the opposite direction. Ideally, these would be from multiple labs, so that different researchers independently get similar findings. Also, having multiple different methods is reduces the danger that it’s all just a quirk of how you measure something.

It’s essential today to have preregistered successful replication, so that would be a second criterion. Third, I would value having some real-world data, even though these are inevitably a bit more confounded, to bolster confidence that it’s not just a lab thing, and that it does apply real phenomena in normal life.

Lastly, there is the new method: multi-site replication. Essentially, one signs up a dozen or so different labs to run the experiment, and results are combined. This has become popular in recent years. I recently wrote a narrative review of all 35 of these that have been published. If we require that the best-replicated findings should have a significant supportive result from a multi-site project, then there are only five candidates.

Three of those candidates are quite narrowly focused: that we look at someone when we’re expecting them to speak; that eyewitnesses who confer about the episode can influence each other’s testimony; and that people imagine they would be more willing to reconcile after getting a message of social acceptance. Although those look good on the multi-lab criterion, they have much less support from the other three criteria. That leaves two other candidates for the best-replicated finding: ego depletion, and two routes to persuasion (the Elaboration Likelihood Model).

Now, I'm not a neutral person here, as I was the author on several of the first ego depletion papers. It is fair to assume I would be biased in its favour. Hence readers should not take my word for it but form their own opinion based on the evidence. If nothing else, being identified with ego depletion research made me more likely to notice information about it, even while immersed in a much bigger project.

Ego depletion has a strong case to be the best-replicated finding in social psychology. Ego depletion is based on the theory that willpower is limited, so that after people expend effort on self-control or decision making, they will perform worse on subsequent tasks that also require self-control or decision making. Dang and colleagues (2021) ran a study in 12 labs that found significant support for this effect.

Ego depletion also looks quite strong based on the other criteria. Several years ago, Friese and colleagues (2019) counted 600 published studies, mostly with significant support (and essentially none significant in the opposite direction). Those included many different labs — indeed on different continents — and different procedures, too. There are even more now, plus many supportive unpublished results. There are pre-registered, large-sample replications (e.g., Garrison et al., 2018). There are plenty of real-world and field-study findings, such as with healthcare workers and athletes.

Two other multi-site replication attempts provided mixed results. An early one by Hagger et al. (2016) failed to test the hypothesis due to adopting a weird procedure. The main manipulation check (fatigue) showed no effect. A secondary check (effort) did show some effect, and although the results were gleefully trumpeted as a failure to replicate, a reanalysis of the data by Dang (2016) found that it actually did support ego depletion, to the very limited extent that the manipulation worked. A more recent attempt by Vohs et al. (2021) was almost perfectly inconclusive. Their exclusion criteria led to throwing out a third of their data, over a thousand participants, which would invalidate most studies. Vohs found a significant effect with the full sample but a nonsignificant one after throwing out so much data. The effect sizes were about the same, so it may be a statistical power issue.

Self-Control Essential Reads

On Hold or in Overdrive: Decoding Inaction and Impulsivity

Does Performing Rituals Build Character?

In any case, given that nearly all multi-site replications in social psychology fail, the standout finding is the positive result by Dang et al. (2021). That’s the important news.

The main competitor for the title of best-replicated finding would be the Elaboration Likelihood Model of persuasion. I’ve always been a big fan of that theory and so if that were to be declared the winner, that’s fine with me. It had one successful (Ebersole et al., 2017) and one unsuccessful multi-lab replication, and I think the success counts more than the failure. The successful one was not pure social psychology, however, because one independent variable was a personality trait measure (and those generally replicate better, I believe). It does well by the other criteria also.

There is a multi-site replication in progress on cognitive dissonance. If that yields a significant result, then that probably would take over first place, because it satisfies the other criteria also. For now, however, I think ego depletion has a strong case to be regarded as the best-replicated finding in social psychology.

The multi-site method has problems that cause false-negative findings, which I’ll address in another post. Perhaps we should accept the fact that much in social psychology requires a level of personal and interpersonal engagement that is often lacking when large groups of people sit at computers and make ratings. Ego depletion will only work insofar as participants put in the effort to deplete their energy. New methods emphasize longer manipulations, such as spending a half-hour performing self-regulating tasks. Some authors have reported failure to replicate ego depletion – but usually because they failed to manipulate the independent variable (as shown by the manipulation check), so they did not test the hypothesis.

I suppose another criterion for "best-replicated finding" might be that nobody has reported any failures to replicate it. In that case, obviously, ego depletion would be ruled out, as would ELM, and even cognitive dissonance. Indeed, having worked the last five years at an Australian university where every faculty member had to read and mark a dozen or more honors (honours) theses every year, one is constantly startled by failures to replicate many basic and standard effects. And if we allow that competence is important to replicating effects, then very plausibly that is true among seasoned researchers also, so unsuccessful replications could happen anywhere. Plus, there is no way to count how many failed studies of any particular effect are in the file drawers and student thesis libraries. I think that the never-failed criterion is unworkable.

Obviously, no one should just take my word for it. But if you wish to make the case for a different finding as social psychology’s best-replicated, please write a brief nomination. Use the criteria mentioned above, or make the argument for different criteria. Publish or at least email to me and I’ll get the word out.

For now, though, ego depletion has a strong claim to be social psychology’s best-replicated finding. I also think there are many well-replicated findings that simply have not been subjected to a multi-site replication attempt. That may be lucky for them since most such attempts fail. Nevertheless, it seems desirable to try to get some of them successfully replicated. The self-serving attributional bias, social loafing, and the provocation-aggression link are strong candidates.