Does Confidence Reflect Accuracy, or Just Consensus?
Some facts that seem perfectly obvious are also incorrect.
Posted May 10, 2018
Quick: Which city is further north, Austin TX or San Antonio, TX? How confident are you that you’re right?
Neither of these questions seems strange to ask; we often answer general knowledge questions. In addition, we often have some degree of confidence that our answer is correct. A natural question is whether our confidence is a genuine indicator of our correctness. That is, should you believe people when they seem absolutely certain that their answer to a question is correct?
This question was addressed in an interesting paper in the May 2018 issue of the Journal of Experimental Psychology: General by Asher Koriat.
He reviews results from 16 different studies that posed a variety of questions for which participants had to select one of two possible answers. Some studies looked at general knowledge questions. Some were perceptual question that involved discerning which of two lines was longer or which of two shapes had a larger area. Some tasks required predicting other people’s responses to questions.
For each study, Koriat divided the questions by whether the majority of people got the answer correct (which he called consensus correct items) or whether the majority of people got the answer wrong (which he called consensus wrong items).
The consensus correct items showed a pattern that is consistent with the proposal that people are more confident in their responses when they are most accurate. The higher the level of agreement among people about which answer was correct, the higher the average confidence ratings of the people for that item.
However, for the consensus wrong items, the higher the percentage of people who believed the wrong answer was correct, the higher the average confidence in their judgments. That is, for items that the majority of people get wrong, confidence also increases with the degree of consensus about the answer.
These findings suggest that accuracy and confidence involve separate sources of information. Accuracy for a particular question reflects the use of some combination of memory, reasoning, and heuristics that allow people to make judgments. For example, you might simply recall that Austin, Texas is about 70 miles north of San Antonio, Texas. For a different question—"Which city is further north, Toronto, Canada or Venice, Italy?"—you might use some reasoning and rules of thumb. For example, you might reason that Toronto is in a country that is north of the USA, while Venice is in a country in southern Europe and so you might guess that Venice is south of Toronto. (You would be wrong, in this case.)
Confidence in a judgment, however, seems to be based on factors related to the consistency of judgments across people. That is, your confidence in a judgment is based on how obvious it seems to you that the answer is correct, and that obviousness is often true for other people as well.
Unfortunately, there are times when there are answers that seem perfectly obvious, but are also incorrect.
Some of these instances involve how often information is reported. You might believe that there is a lot of scientific evidence that genetically modified foods (GMOs) are dangerous given the relationship between non-GMO labels and supermarkets that cater to people seeking healthy food. Yet there is very little scientific evidence that GMOs are bad for you.
In other situations, the reasoning process that leads to the wrong answer seems all too easy, as in the Toronto/Venice example.
In the end, it is important to realize that confidence may be more a marker of how likely other people are to agree with a response than whether that response is objectively true.
Koriat, A. (2018). When reality is out of focus: Can people tell whether their beliefs and judgments are correct or wrong? Journal of Experimental Psychology: General, 147(5), 613-631.