How to Avoid Being Deceived by Opinion Polls

Many polls are intended to shape your ideas and behavior

Posted Nov 23, 2015

According to a headline on Cosmopolitan magazine’s website, “Survey Shows That Most Americans Think Caitlyn Jenner Is Going to Change the World.”  I’ll lay odds that most Americans would be surprised to learn that this is what they think.  Take me, for instance.  I’m of the opinion that Caitlyn Jenner has done more than her fair share of changing, thank you very much.  Does this mean I am not a "mostest" American?

Actually, my article is not about Caitlyn Jenner, bless her heart (as we Southerners say).  It’s about fallible polls and the twisted interpretations of same as frequently reported by the media.  Whoever wrote the above headline either knew it was hype or didn’t know and didn’t care.  I don’t know which is worse.  Maybe they both are (worse, that is).  There’s nothing in the poll about Jenner changing the world.  At best, it suggests that she may help boost acceptance of transgenderism.

The poll was conducted jointly by NBC News and the University of Pennsylvania.  Sounds credible, right?  That’s what you’re supposed to think.  The purpose of this poll is to influence your thinking.  The not-so-subtle implication is that you’re in the backward-thinking minority if you don’t agree.  But why this poll, on this topic?  And why now?  What are they selling?  Those are the kinds of questions that provide a basis for testing the merits of this or any other poll.

If a survey is to be used to represent the opinions of a majority of Americans (or any other large group), it must be conducted using scientifically sound methodology.  Otherwise it represents nothing more than the opinions of those who participated in the survey.  No poll is absolutely pristine and flawless.  There is always a margin of sampling error or bias.  We need to see if we can find the flaws and determine if they are of minimal or major importance.  Let’s ferret out some clues.

The first clue is very subtle – like a baseball bat impacting the bridge of your nose at warp speed.  The survey was conducted using a non-probability sample.  In plain English, the participants in the survey were not randomly selected. Sometimes this approach is necessary because random sampling would be too arduous or expensive.  There are various types of non-probability sampling, some better than others for this kind of poll.  The pollsters used convenience sampling – i.e., they relied on people who volunteered to participate. 

Convenience sampling undermines the claim that most Americans agree with the results.  People who volunteer to participate may have a personal agenda, a biased perspective, an emotional connection to the issue, or may be responding simply because they were promised a reward for participating. (I have no information as to whether NBC News/UPenn offered a reward.)  Suppose you were going to conduct a poll on Americans’ views on raising the minimum wage.  Would you trust the results if the only people surveyed were those who volunteered to give their opinions on that topic?  To claim that the results of such a poll would represent the views of most Americans would be farfetched.

As is expected practice in academic research, the UPenn pollsters acknowledge these limitations in their methodology:

Because the sample is based on those who initially self-selected for participation rather than a probability sample, no estimates of sampling error can be calculated. All surveys may be subject to multiple sources of error, including, but not limited to sampling error, coverage error, and measurement error.

Another weakness is that survey responses were collected via online polling (i.e., via Survey Monkey).  There is nothing wrong with online polling as a means of collecting data.  The problem lies in the fact that older people, especially those over 70, are less likely to respond to an online poll.  They may therefore be under-represented in the sample, and this again undermines the claim that the poll is representative of most Americans.

Since this poll was commissioned by NBC News and has been distributed to other media outlets, such as Cosmopolitan, the reasonable assumption is that it was commissioned for the purpose of making news and boosting audience share and click rates.  (Oh, for those halcyon days when the media didn’t make the news, they just retailed it.)

The UPenn pollsters did about as well as could be expected, given the scope of their task and the constraints of time and budget.  However, an imperfect research design yields imperfect results.  These are certainly not results that support the claim that “most Americans think that Caitlyn Jenner is going to change the world.”

World to Caitlyn:  Don’t go changing.