Why Normality Matters: Insights From a Meta-Analysis
Summarizing how exceptional circumstances affect our cognition and behavior.
Posted Jan 24, 2021
This post was written by Adrien Fillon from the University of Aix-Marseille, France, who led a meta-analysis of the "exceptionality effect" discussed in this post. Gilad Feldman, the corresponding author on the meta-analysis, edited this post for Psychology Today.
Imagine a common situation in which you are leaving work to go back home and are faced with a choice between two routes: your usual routine road that you typically take every day, or a different less-traveled road.
Now let's imagine that after having made that choice you face the unfortunate outcome of experiencing a car accident while driving back. Try and think, would the road you chose to drive on impact how you feel about the accident? Would you experience stronger regret over the negative outcome if you took the usual road or if you took the unusual road?
Norm theory and the exceptionality effect
In 1986, Daniel Kahneman and Dale Miller published an article called "Norm theory: comparing reality to its alternatives". Their goal was to understand the concept of "normality" and its impact on emotions, cognition, and behavior.
In presenting their participants with the scenario we described above, they found that participants found an unfortunate outcome following a decision to drive on the exceptional road to be more regretful than the same outcome following a decision to drive on the routine route. They concluded that exceptionality, deviations from a reference point, and "normality" impact our judgments, decision-making, emotions, and cognitions. There have been many demonstrations of the exceptionality effect. For example, being the victim of a shooting in a convenience store either frequently visited or rarely visited. Or consider a person being robbed by a hitchhiker, if the behavior follows or deviates from a routine giving hitchhikers rides.
Exceptionality effect meta-analysis
We conducted a systematic review of the exceptionality effect literature and found 51 experiments conducted looking at the effects of past behavior exceptionality, the comparison to previous established behavior, one of several types of normality reference points.
We found that, when we are faced with a negative outcome, we tend to attribute stronger regret when the behavior associated with that outcome was exceptional than when it was normal. Exceptionality tends to more easily elicit counterfactual thoughts, thoughts of alternative realities of what might have been "if only." In exceptional circumstances, it is easy to think of an alternative reality of simply acting normal—so that, for example, in the scenarios discussed above one might think, "If only I have taken the normal route, I would have avoided the unfortunate outcome."
The effect also goes beyond regret and counterfactuals, as in cases where harm was involved people also rely on exceptionality cues in their attributions of blame and associated decisions regarding punishment (for offending agents) and compensation (for the offended agents).
We further examined which factors might strengthen or weaken the effect. We found that the exceptionality effect was stronger in studies where exceptionality was aligned with action, than when it is aligned with inaction. We also attempted to examine new directions, such as examining the controllability, severity, and outcome rarity factors, yet more research is needed before we can provide a conclusive meta-analyzed summary.
To read more about the meta-analysis, see: Fillon, A., Kutscher, L., & Feldman, G. (2020). Impact of past behaviour normality: meta-analysis of exceptionality effect. Cognition and Emotion, 1-21.
Meta-analysis: The process
Lucas Kutscher first completed the meta-analysis together with a replication and extension of the exceptionality effect as part of his one-year master's thesis project. The two components were then separately submitted to journals, and Lucas graduated and moved on to work in the industry. The replication and extension was published smoothly, yet the meta-analysis required more work following an invitation to revise and resubmit, and Gilad was seeking scholars to join working on the meta-analysis to address the reviewers' concerns and bring this to conclusion. After my successful collaboration with Gilad on a different project examining the exceptionality effect in the context of free will attributions (Fillon et al., 2019), I joined the meta-analysis project.
How difficult was it to conduct a meta-analysis?
Taking over work the of others is always a challenge, but in this case, Lucas and Gilad carefully annotated and commented the datasheet and code to make it easily reproducible. It helped me understand the process of meta-analysis and gave me new tools to improve my skills in open-science and reproducibility.
I was familiar with the exceptionality effect and associated literature but was lacking experience with the actual process of conducting a meta-analysis. Fortunately, there are many resources to quickly learn how to perform meta-analyses. We have access to free books, easy modules for Jamovi (MAJOR), and many resources and tutorials over the internet (see, for example, this brilliant book to easily conduct a meta-analysis in R). Also, the mentality has changed. We now have much more possibilities to preregister meta-analyses, and registered reports are far less exceptional as it was in the beginning of this meta-analysis. We have more templates and guidelines (PRISMA, NIRO, etc.), and they are used more frequently. Meta-analysis can serve both as a tool to aggregate effects and estimate the overall effect size, as well as a tool to assess and improve science.
I worked long and hard to address the major revision and finally completed the revision, leading to the successful publication of the article two-and-a-half years after it was first initiated.
Implications of exceptionality effect meta-analysis for researchers
The first implication is that we found large differences between different study designs, between-subjects, within-subjects, or one-condition comparison. In within-subjects design in exceptionality effect experiments, every participant answers all scenarios in all conditions in randomized order, and for each scenario answers questions about regret, responsibility, counterfactuals. In between-subjects design, each participant is exposed only to one condition, and in exceptionality effect experiments it is either the routine condition or the exceptional condition. In the one-condition comparison design, participants are asked to compare protagonists representing the two conditions, exceptionality versus normal, for example: Which protagonist feels stronger regret—the one who acted as routine or the one that acted exceptionally?
We found out that in the comparison design, the size effect was up to four times higher than for the between-within designs, suggesting that the forced comparison may lead to an inflated effect.
The second implication is about the importance of open data in the era of meta-analysis. One reviewer, Joe Hilgard (who decided to sign his review) went through our code and dataset and helped us improve our code. We also shared the preprint on OSF and Researchgate, and one helpful reviewer, Michał Białek found and reported an error in our figures. As always, sharing the preprints, preregistration, code, and datasets improved the quality of our work.
Finally, this meta-analysis served as the basis for a template used in other meta-analyses projects our team conducted in the field of judgment and decision-making (Yeung et al., 2020a), but also in commitment theory (Fillon et al., 2020b), and in cognitive dissonance (Jaubert et al., 2020). Our goal is to try and standardize open and reproducible meta-analyses by providing easy templates for manuscripts and code. We developed a template that includes a main manuscript, supplementary, coding/search datasheet, and Rmarkdown analysis code, for either experimental meta-analysis (Yeung et al., 2020b) or correlational meta-analysis (Fillon et al., 2020c). We hope that these templates would help researchers do better more open meta-analyses in a Registered Report format.
Regarding exceptionality, we so far conducted two successful replications of the original exceptionality effect study by Kahneman and Miller (1986). In the first, we extended the findings to morality (Fillon et al., 2019) and in the second to social norms and perceived luck (Kutscher & Feldman, 2018). Yet there is much that is left to be explored: What is the relationship between norms, emotions, and morality? What are the moderators for the exceptionality effect: under what conditions is the effect weakened or strengthened? Researchers can build on this meta-analysis to explore exciting new directions about this classic phenomenon.
Reflecting on my journey
I've also recorded a video where I reflect on this collaboration, covering some of my work on the exceptionality meta-analysis:
Meta-Analysis Registered Report Templates
Our team has been working on templates for a meta-analysis Registered Report. To learn more about Registered Reports, and how they are related to and are different from pre-registrations, please see our workshop "Pre-registrations and Registered Reports | Open Science workshops 2020 | Gilad Feldman."
We would welcome your feedback, and you are welcome to go ahead and make use of those in your own meta-analyses.
Experimental Meta-Analysis Registered Report Template (led by Siu Kit Yeung)
Correlational Meta-Analysis Registered Report Template (led by me, Adrien Fillon)
Fillon, A.A., Lantian, A., Feldman, G., & Ngbala, A. (2019). Exceptionality Effect in Agency: Exceptional Choices Attributed Higher Free Will Than Routine. Center for Open Science. https://doi.org/10.31234/osf.io/sep8f
Fillon, A., Kutscher, L., & Feldman, G. (2020a). Impact of past behaviour normality: meta-analysis of exceptionality effect. Cognition and Emotion, 1–21. https://doi.org/10.1080/02699931.2020.1816910
Fillon, A., Pascual, A., Souchet, L., Girandola, F. (2020b). The effectiveness of the “But-you-are-free” technique: Meta-analysis and re-examination of the technique. [Manuscript in preparation]. https://doi.org/10.31234/osf.io/3ds26
Fillon, A., Yeung, S. K., Feldman, G., & Xiao, Q. (2020c). Correlational Studies Meta-Analysis Registered Report Templates. [Manuscript in preparation].
Kahneman, D., & Miller, D. T. (1986). Norm theory: Comparing reality to its alternatives. Psychological Review, 93(2), 136–153. https://doi.org/10.1037/0033-295x.93.2.136
Kutscher, L., & Feldman, G. (2018). The impact of past behaviour normality on regret: replication and extension of three experiments of the exceptionality effect. Cognition and Emotion, 33(5), 901–914. https://doi.org/10.1080/02699931.2018.1504747
Yeung, S. K., Feldman, G., Fillon, A., Protzko, J., Elsherif, M. M., Xiao, Q., & Pickering, J. (2020). Experimental Studies Meta-Analysis Registered Report Templates. [Manuscript in preparation].
Jaubert, S., Fillon, A. A., Souchet, L., & Girandola, F. (2020, December 4). Vicarious Dissonance: Meta-analysis registered report. https://doi.org/10.31234/osf.io/hc2n3