5 Tips for Creating a Psychological Survey

How to get the most out of your psychological surveys

Posted Oct 09, 2018

TeroVesalainen / Pixabay
Source: TeroVesalainen / Pixabay

These days, there is a survey for pretty much anything. How likely are you to vote? How much to do you support thus-and-such candidate? How effective was the customer service you received from our business? How prone are you toward anxiety? How likely would you be to spend money on some item? How effective was that teacher in delivering the course content? And more …

All kinds of people are in the business of measuring psychological variables. What few people realize is this: Creating a good psychological measure is not as easy as it might seem.

In fact, from the perspective of someone who has created all kinds of measures of a broad array of psychological concepts over the years, I can say with confidence that most people miss the boat when developing surveys. An education in psychometrics (psychological measurement) and/or psychological research methods, in fact, goes a long way toward helping people make better measuring instruments - whether you are creating a scale to measure some psychological traits or you are asking customers at a bank how satisfied they are with some product. And everything in between.

Below are five tips to help you make better surveys in your work.

1. Put items on the same scale as one another.

If you have a bunch of items that measure satisfaction with some product, for instance, you should put all of the items on the same scale as one another. For instance, you might have three items that are all on a 1-5 scale, with 1 corresponding to strongly disagree, 3 being neutral, and 5 being strongly agree. Using such a scale, you can then mold all kinds of items about your product in a format that works with this scale. For instance, you can ask people to rate the degree to which they agree with the following three items:

  • I am happy that I purchased the product.

Strongly Disagree 1 2 3 4 5 Strongly Agree

  • I would recommend the product to a friend.

Strongly Disagree 1 2 3 4 5 Strongly Agree

  • The product worked as it was advertised.

Strongly Disagree 1 2 3 4 5 Strongly Agree

If all the items are on the same scale as one another, you can do all kinds of things such as add scores together across items to create a total score, compare average scores across each of the items directly with one another, etc.

2. Reverse-score some items.

Not everyone loves taking a survey. Sometimes, people try to fly through a survey carelessly. One way to address this point is to reverse-score some items. This is actually a simple idea. All it means is that for a subset of items, what corresponds to a high score will be coded the opposite, numerically, is it is for other items. Using the product satisfaction measure that we are working with in the prior section, this would just mean adding some items where high scores corresponded to negative attitudes about the product and low scores corresponded to positive scores about the product. For instance, you could add these two items:

  • I found the product to be very low in quality.

Strongly Disagree 1 2 3 4 5 Strongly Agree

  • I am sorry that I purchased the product.

Strongly Disagree 1 2 3 4 5 Strongly Agree

The one thing you’d need to keep in mind if you include reverse-scored items is that later on, if you sum together scores across items, you would need to recode the items from the reverse-scored items. So in the current case, for instance, before adding up the scores for each of the items to get an overall sense of a particular customer’s attitude about the product, you would work with the reverse-scored items and convert 1 to 5, 2 to 4, 4 to 2, and 5 to 1 (leaving 3 as 3). This would make sure that all scores are arranged in the same direction quantitatively. Importantly, this recoding process is done at the data-analysis stage (after you have collected all the data).

janjf93 / Pixabay
Source: janjf93 / Pixabay

3. If you can measure a variable as a continuous variable, you should do so.

Variables come in all kinds of varieties. And sometimes you can measure the same variable in multiple ways. For instance, in terms of one’s socioeconomic status, we could do any of the following:

  • Ask people for their annual household income in dollars by filling in a blank.
  • Ask people to select one of various ranges of household income (e.g., 0-$30,000; $31,000-$60,000; etc.)
  • Divide people, based on a cutoff of an annual household income of $80,000 as “rich or poor.”

The first option (A) would be what we could call a continuous variable - measuring the variable with as many possible degrees of differentiation across scores as possible. The second option (B) would be an ordinal variable - measuring this variable in terms of the order that someone’s income comes in within pre-set categories (lowest, second lowest, third lowest, etc.). The final measure, which would be terrible, by the way, would be a categorical variable - simply dividing people into two simple categories.

To really get into the details, you would need to take a course in statistics (or you can read Sara Hall and my book, Straightforward Statistics) But the short version is this: If you can use a true continuous measure of some variable, you should do so as doing so provides you with the most possible information. And the statistics that we use for continuous variables are more powerful than are statistics used for these other kinds of variables. In other words, even if option B (above) seems best, it's not. Choose option A. Choose continuous measures of your variables when at all possible.

4. Avoid double-barrelled items, which ask two different questions.

It is very common for people to include items in surveys that ask about two different things. For instance, in a survey about the product used as the example here, imagine an item as follows:

  • The product is really great and is very affordable.

Strongly Disagree 1 2 3 4 5 Strongly Agree

See the problem? Some products are truly great but are not truly affordable (think about a Ferrari!). The quality of the item is, conceptually, discrete from its affordability. And the questions in your survey should reflect this fact. Otherwise, you can’t really be sure how people are interpreting your question. Try to make each discrete item in your survey have a singular, unambiguous meaning.

5. Think about how you will use the data at every step of survey development.

When people make a survey of some kind, they often get excited. And I don’t blame them. It is a fun process! This said, it’s very important to always think about how each item in your survey is going to be used. Should you ask participants to describe their parents’ level of education? Should you ask them to name what geographical region they live in? Should you ask them if they have kids? If they are married?

The answer to each of these questions is this: It depends on what you are trying to accomplish. Kitchen-sinking is often a great way to ruin an otherwise good survey. Try to keep your measure short and sweet - and only include items that you have a strong rationale for. If you have no reason to ask about participants’ marital status, and you have no specific plans to analyze the data that you’d get from such an item, then do everyone a favor and don’t include it. Your survey should be targeted and fully driven by whatever led you to create the measure in the first place.

Bottom Line

Creating psychological measures is one of these things that seems easier than it is. To do this well, there is actually quite a lot of knowledge related to statistics and research methodology that you need to know. The five points described here should serve as a good start. Good luck with your survey creation. And always keep in mind that the people completing your survey are actual humans, just like you and me. Creating efficient, streamlined, high-quality surveys reduces the likelihood that you will be wasting people’s time.