Can We Estimate Our Own Ability to Reason?
The Dunning-Kruger effect isn’t the only snag for self-report reasoning scales.
Posted July 20, 2021 | Reviewed by Ekua Hagan
- Self-report measures of reasoning rely on people's perceptions of their own reasoning abilities, while behavioral tests show how they perform.
- Research reveals a discrepancy between self-reported and behavioral tests of reasoning.
- To understand people's reasoning ability, both self-reports and behavioral measures are needed.
Suppose you ask me how often I tend to overcome faulty intuitions and biases by stopping to reflect on my initial impulse. I probably think that I'm relatively reflective. I've been studying philosophy for over a decade and philosophers tend to be reflective (Livengood et al., 2010). Also, I've been doing cognitive science for nearly a decade and cognitive scientists are probably familiar with the evidence suggesting that we should question our gut in at least some circumstances (e.g., Scherer et al., 2017; Tversky & Kahneman, 1983).
But suppose you also give me a reflection test with questions that lure me toward intuitively appealing answers that—upon reflection—I can realize are incorrect (e.g., Frederick, 2005). That may reveal that I am not as reflective as I believe. This would make sense. We often overestimate ourselves (Zell et al., 2020).
However, there is more: Some have found that the less skilled people are, the less they seem to realize it (Kruger & Dunning, 1999). So if I am a less reflective reasoner, then I may be even more likely to overestimate my reflective tendencies than more reflective reasoners.
Self-Report vs. Behavior
This discrepancy between self-reported and behavioral tests of reasoning raises questions about the trustworthiness of self-reported reasoning ability. After all, if we measure people's reasoning ability the way we measure many personality traits—e.g., by merely asking people to rate their agreement with statements like, "I prefer complex to simple problems" and "Thinking is not my idea of fun" (Cacioppo & Petty, 1982)—then we may be systematically overestimating poor reasoners who are more likely to overestimate their reasoning ability.
However, there may be a bigger problem than the possibility that poor reasoners are less aware of their actual reasoning performance. The bigger problem may be that self-reported habits can be a poor predictor of actual habits more generally (a la Parry et al., 2021).
Various studies have found this kind of disconnect between self-reported and behavioral measures of reflective reasoning.
- Mirja Peret and Petra Rekat (2020) had high school students and professionals (a) report their agreement with sentences like the ones above (a.k.a., the "Need for Cognition" or NFC scale) and (b) complete a series of financial decision-making tasks. Alas, they found "no difference in the NFC scores among participants that solved [financial decision-making tasks] correctly and those who [did] not” (ibid.).
- Gärtner et al (2021) had people complete the NFC items as well various measures of cognitive control, working memory, and other tests of executive function. They "found no conclusive evidence that NFC was related to any executive function measure [but instead] obtained ...moderate evidence for the null hypothesis" (ibid.).
- Newman and colleagues (2020, Experiment 3) found that people who scored themselves higher in "need for cognition" were more (not less) "susceptible to the illusory truth effect": the bias toward believing things that are repeated—even if they are false. (Then again, De Keersmaecker et al. (2019) found that even behavioral measures of reasoning didn’t seem to predict less susceptibility to the illusory truth effect, suggesting that the illusory truth effect may not be as related to reasoning ability as we might have expected.)
- Coutinho et al. (2021) had people score themselves on how unreflectively they reason by reporting their agreement with statements like, "The ﬁrst idea is often the best one" and "When I need to form an opinion about an issue, I completely rely on my intuition"—a.k.a., the Faith In Intuition scale (Epstein et al., 1994). Coutinho and colleagues also had people complete a behavioral test of whether people stop to reconsider initial intuitions. Not only did they find a Dunning-Kruger effect, they also found that self-reported "faith in intuition" was "not related to intuitive responding" on a behavioral test, replicating earlier work (Pennycook et al., 2015).
- One potential takeaway is that measuring reasoning via self-report is inferior to measuring reasoning via actual behavior. After all, worse reasoners have been more likely to overestimate their reasoning skills and some self-scored measures of reasoning seemed to be unrelated to actual reasoning performance.
- Another possible takeaway is that self-reported and/or behavioral measures of reasoning do not measure what we think they do. Perhaps one's self-reported habits measure idealized perceptions of oneself rather than one's actual tendencies (e.g., Brown-Iannuzzi et al., 2019). Or perhaps some behavioral reflective reasoning tests measure more than just reflection (e.g., Byrd & Conway, 2019). These possibilities could help explain why we find less overlap than expected when we compare behavioral measures of reasoning with self-report measures of reasoning.
Of course, these possibilities are not mutually exclusive. The point is just that if you need to assess someone's reasoning, then you might want to triangulate with both self-report and behavioral measures—especially if you are evaluating your own reasoning ability.
Brown-Iannuzzi, J. L., Najle, M. B., & Gervais, W. M. (2019). The Illusion of Political Tolerance: Social Desirability and Self-Reported Voting Preferences. Social Psychological and Personality Science, 10(3), 364–373. https://doi.org/10.1177/1948550618760147
Byrd, N., & Conway, P. (2019). Not all who ponder count costs: Arithmetic reflection predicts utilitarian tendencies, but logical reflection predicts both deontological and utilitarian tendencies. Cognition, 192, 103995. https://doi.org/10.1016/j.cognition.2019.06.007
Cacioppo, J. T., & Petty, R. E. (1982). The Need For Cognition. Journal of Personality and Social Psychology; Journal of Personality and Social Psychology, 42(1), 116. https://doi.org/10.1037/0022-35126.96.36.199
Coutinho, M. V. C., Thomas, J., Alsuwaidi, A. S. M., & Couchman, J. J. (2021). Dunning-Kruger Effect: Intuitive Errors Predict Overconfidence on the Cognitive Reflection Test. Frontiers in Psychology, 12. https://doi.org/10.3389/fpsyg.2021.603225
Frederick, S. (2005). Cognitive Reflection and Decision Making. Journal of Economic Perspectives, 19(4), 25–42. https://doi.org/10.1257/089533005775196732
Gärtner, A., Grass, J., Wolff, M., Goschke, T., Strobel, A., & Strobel, A. (2021). No relation of Need for Cognition to basic executive functions. Journal of Personality. https://doi.org/10.1111/jopy.12639
Kruger, J., & Dunning, D. (1999). Unskilled and unaware of it: How difficulties in recognizing one’s own incompetence lead to inflated self-assessments. Journal of Personality and Social Psychology, 77(6), 1121–1134. https://doi.org/10.1037/0022-35188.8.131.521
Livengood, J., Sytsma, J., Feltz, A., Scheines, R., & Machery, E. (2010). Philosophical temperament. Philosophical Psychology, 23(3), 313–330. https://doi.org/10.1080/09515089.2010.490941
Newman, Eryn. J., Jalbert, M. C., Schwarz, N., & Ly, D. P. (2020). Truthiness, the illusory truth effect, and the role of need for cognition. Consciousness and Cognition, 78, 102866. https://doi.org/10.1016/j.concog.2019.102866
Parry, D. A., Davidson, B. I., Sewall, C. J. R., Fisher, J. T., Mieczkowski, H., & Quintana, D. S. (2021). A systematic review and meta-analysis of discrepancies between logged and self-reported digital media use. Nature Human Behaviour. https://doi.org/10.1038/s41562-021-01117-5
Pennycook, G., Cheyne, J. A., Koehler, D. J., & Fugelsang, J. A. (2015). Is the cognitive reflection test a measure of both reflection and intuition? Behavior Research Methods, 1–8. https://doi.org/10.3758/s13428-015-0576-1
Perat, M., & Rekar, P. (2020, December 26). Need for cognition is a poor measure of analytical thinking. 9th Economic and Business Review Conference & SEB LU Doctoral Conference 2020.
Scherer, L. D., Yates, J. F., Baker, S. G., & Valentine, K. D. (2017). The Influence of Effortful Thought and Cognitive Proficiencies on the Conjunction Fallacy: Implications for Dual-Process Theories of Reasoning and Judgment. Personality and Social Psychology Bulletin, 0146167217700607. https://doi.org/10.1177/0146167217700607
Tversky, A., & Kahneman, D. (1983). Extensional versus intuitive reasoning: The conjunction fallacy in probability judgment. Psychological Review, 90(4), 293–315. https://doi.org/10.1037/0033-295X.90.4.293
Zell, E., Strickhouser, J. E., Sedikides, C., & Alicke, M. D. (2020). The better-than-average effect in comparative self-evaluation: A comprehensive review and meta-analysis. Psychological Bulletin, 146(2), 118–149. https://doi.org/10.1037/bul0000218