If you turn on the TV these days, you're bound to hear the results of polls predicting aspects of the election. Some of these polls are done by polling organizations, but these days the websites from many news outlets also contain surveys that people can answer. The results from these surveys are often presented as part of cable news broadcasts followed by the standard disclaimer, "Of course, these results are unscientific."
The internet has made it possible to gather the opinions of large numbers of people who are willing to take a few seconds out of their day to answer a question. It is not uncommon for an internet poll on a major website like CNN to have half a million or more votes cast. At first, access to thousands of internet users seems appealing. It would take hundreds of person-hours to contact this many people, and yet suddenly CNN has hundreds of thousands of votes.
Unfortunately, there need not be strength in numbers. The disclaimer "these results are unscientific," is actually a euphemism for "these results are inaccurate." What makes these surveys inaccurate? Presumably, organizations take surveys because they want to know how a particular population feels about a specific issue. In order to estimate what a particular population feels about an issue, you must get an unbiased sample from the population. There are many techniques for generating an unbiased sample, and polling organizations have gotten quite good at obtaining such samples.
Assume that a news organization is interested in the attitudes of people in the United States. Internet surveys will get a biased sample from this population. The only people who respond are those people who own computers, happen to be surfing the net, and decide to answer the question on the survey. Not all people own computers. Not all computer-owners surf the net with great frequency, and not all web-surfers will necessarily answer a particular question when they see it. Perhaps only those with an axe to grind will bother to answer a particular question. Indeed, I received emails from groups interested in the campaign suggesting that I go vote in polls on major news websites following the debate.
Thus, the results of most internet surveys need not bear any resemblance to the actual beliefs of the assumed target population. So, why does this matter? After all, news reporters on television always give the standard disclaimer, "these results are not scientific" whenever they present poll results. If they are being honest about the value of the surveys, then who is being hurt?
The problem lies in the psychology of human judgment. Much research going back to the classic work by Kahneman and Tversky demonstrates that when people make a judgment about some topic with which they are not expert, they tend to use a strategy called "anchoring and adjustment." First, people access a belief (which serves as the anchor) and then adjust that belief to take into account additional factors. This strategy is remarkably effective in most situations. It can be flawed in situations in which the anchor is inaccurate, or in which they fail to adjust properly.
Internet survey results that are broadcast as part of the news become the source for anchors that can be used in judgments. As the news people point out when the story is broadcast, these results are inaccurate, and so they form a poor basis for judgments. Unfortunately, while human memory is good at making relevant information accessible, it is poor at marking that information as inaccurate. Thus, people will use these survey results as the basis for judgments about the way people feel about an issue, despite the fact that there is no good reason to believe they are an accurate measure of the attitude in the population as a whole. Even if they do recognize that the survey is inaccurate, they will start with these results as an anchor and then attempt to adjust the number to make it more accurate. Without knowing anything about the sample of people who responded to the survey, it is not possible to adjust the survey results to make them more accurate.
If news organizations knowingly broadcast false information about the beliefs of a major corporation, they would be sued. Why, then, do we all sit back and allow these false statements to be aired regularly with only a flimsy disclaimer?
