What Psychologists Shouldn't Be Studying

Can psychologists take their own advice to improve their science?

Posted Jul 12, 2020

 Photo by Lisa Fotios from Pexels
Not all measurements are relevant.
Source: Photo by Lisa Fotios from Pexels

I shouldn’t have been surprised by the response, but I was. A few years ago, I attended a panel discussion with a set of famous social psychologists debating what to do about the credibility revolution (or what they would have then called “The Replication Crisis”). Several prominent social psychology experiments had failed to replicate. If you set up the same experiment again, using all the instructions from the original paper, and you couldn’t get the original effect, what did that mean?

Maybe it didn’t make sense to conclude that the larger theoretical point was something general that applied in the wider world. To take one example: If you couldn’t get your explanation for willpower to work consistently in a situation you carefully curated, why should we think it would apply to people trying to exercise willpower in other areas of their everyday life?

The panelists all seemed in agreement about one point: We needed to be more careful about what we measured. This was in response to people whose models had failed replication tests, but who instead blamed the specifics of the measures being used. Finding that one method didn’t work consistently didn’t mean that other methods wouldn’t. Perhaps, the panelists concluded, we needed to focus on really nailing down the methods that worked. Instead of studying willpower with dozens of different types of experiments, we might want to hone in on a few and really understand the details of setting these experiments up right. Then I stood up to ask a question, trying to see if I understood what they were proposing.

 Photo by cottonbro from Pexels
By continuously creating new measures, psychologists never fully understand any measure.
Source: Photo by cottonbro from Pexels

The current social upheaval in the U.S.—due to the COVID-19 pandemic and frustration with political leadership and police violence—has led me to reflect on this discussion again. Many of the foundational experiments in social psychology were explicit attempts to address pressing social issues. Stanley Milgram’s “shock” experiments were an explicit response to Nazi soldiers defending themselves after the Holocaust by saying they “were just following orders.” Latané and Darley’s bystander effect experiments were a response to a news report that numerous New Yorkers had heard the violent murder of Kitty Genovese going on outside their apartments and failed to help (although later historical research found that some neighbors did try to help).

There are many pressing issues in contemporary U.S. society that have social psychological factors at their core: determining which medical advice to trust, finding ways to disagree and compromise civilly, identifying policies and tactics that can lead to safe and effective police activity when it is needed. So what does this have to do with a years-old abstract discussion of scientific reform at a mid-sized academic conference?

At the same academic conference where the panel was discussing the idea of being more thoughtful about how to measure important psychological constructs, a new assistant professor was giving an invited lecture on his research on cross-cultural differences. Invited as a “rising star” of social psychology, he described his seminal work measuring how people navigated coffee shops and bookstores where chairs were too close together. Did they move the chairs apart, or did they turn themselves to the side to squeeze between them? Individualists (like most Americans) were more likely to move the chairs apart. Collectivists (like most of the Chinese people he had studied) were more likely to leave the chairs in place and squeeze through. In the subsequent on-stage interview, he smiled describing how positively people responded to his chair experiments. They could really connect with and understand cross-cultural differences—especially along the dimension of individualism versus collectivism—better after hearing about the chairs.

Another “rising star” invited to speak that day was Sylvia Perry, who studied the way that white parents talk to their children (or don’t) about race. She was measuring at the physiological level—using heart rate and skin conductance measures—the stress felt during important conversations in children’s development. (This isn’t to shame white people but to help provide science-tested tools to parents who want to talk to their children about race.)

Let me now paraphrase the question I asked the panel of social psychologists at the conference. “We’ve just been saying that it takes a lot of effort to understand any one experimental set-up, and so we should use fewer of them and learn how to get them to work in greater depth. If we are only going to use a few specific measures of a construct (like willpower), then shouldn’t we make sure they are the most important and socially relevant ones? Shouldn’t we stop studying how far apart chairs are?” (In this case, I was talking about a different experimental set-up, where how close people pull their chairs to someone are used to measure their feelings of closeness.)

I don’t remember if he used “young man” explicitly in his answer, or if it was just implied in his tone. But the response I got from a senior researcher who had just moments ago been arguing for limiting the experimental setups we use was: “Back in my day, we called distance between chairs a ‘behavioral measure.’” (This is where “young man” may or may not have been said.) For context, ‘behavioral measure’ means something that goes beyond pressing a button on a computer or checking a box on a survey, and psychologists generally agree that we need to use more of them—and to move away from so many clicks and checkmarks. The argument was that anything psychologists measure that goes beyond clicks and checkmarks was obviously good. It didn’t matter if it was something as trivial as distances between chairs.

 Photo by cottonbro from Pexels
Even in physics, abstract concepts need to be "fit out" carefully in experiments.
Source: Photo by cottonbro from Pexels

I think the senior social psychologists were right when they talked about limiting what we measure. This is also an important point made by Philosopher of Science Nancy Cartwright about the way other sciences (like physics) work. Abstract principles always need to be “fitted out” to the specific experimental situation a researcher is using, and that fitting out takes a lot of thought and energy. Psychology does need to limit our scope and dig in more if it wants to make valid and reliable scientific findings. Given that we need to limit ourselves, what should psychologists study?

Psychology should return to the sources of inspiration for the field—the desire to understand social issues with immediate, real-world implications. If we can pick only a few instances of willpower to study, we should pick ones that matter to society: the ability to withstand criticism without starting a campaign of internet bullying or the ability to resist introducing a gun or other weapon into an otherwise routine traffic stop. We need more Sylvia Perry’s: people doing the careful, expensive, and hard work of actually measuring things we really care about.