When to trust scientific predictions—and when to ignore them.
Posted June 28, 2019
One of the key reforms of the Credibility Revolution in psychology research is the use of preregistration: Scientists write down what they predict will happen in an experiment beforehand, so there’s a record which other scientists and the public can look at when evaluating their work. From the outside, it probably seems strange that a large group of scientists has only embraced the idea of making firm predictions ahead of time in the last decade or so. Shouldn’t scientists be making predictions all the time based on our best theories? Doesn’t science progress by testing whether these predictions are right or wrong?
Unfortunately, in psychology, I and many other young researchers were taught a model of manuscript writing based on a famous book chapter by psychologist Daryl Bem called “Writing the Empirical Journal Article.” In it, Bem writes that scientists can write the article they “planned to write when [they] designed [their] study” or the article “that makes the most sense now that you have seen the results” (p. 2). Over the course of the chapter, Bem argues that you should write the article around the most interesting results you can find once you have analyzed your data, minimizing or abandoning what your experiment was originally designed to test if needed.
More thoughtful and prominent psychologists than me have already written about how this can undermine the scientific process, especially when the researcher then writes up the new result as if it was the main point of their study all along. This is referred to as HARK-ing, or Hypothesizing After the Results are Known (HARK), and it is deceitful. It misrepresents results that were used to come up with a theory as results that confirm it. It’s like shooting at the barn door and then drawing a bullseye around wherever you hit.
But something has always nagged me about this argument about Bem’s article. If a psychological theory is so flexible that someone can rewrite their entire manuscript to fit a fluke or unexpected result, why should I care what theory says? If a theory can be made compatible with almost any result, how could I have ever used it to make predictions in the first place?
Imagine I predicted that being put in a good mood, as opposed to a bad mood, makes you focus on big-picture conceptions of the world—you think globally. But if I got the opposite result, I would be happy to explain that being put in a good mood makes you count your blessings, focusing on the details. What am I really doing here, as a researcher? Even if I were to write down a prediction ahead of time, and really commit to the idea that I expect the global focus as opposed to the detail-oriented focus, am I really testing the theory?
In situations like this, I would argue that you should not care about my prediction. You, therefore, should not care about my preregistration. Essentially what I am doing as a researcher is asking: “What happens to people’s attention when I change their moods?” There isn’t any real theory to predict what I’m observing. I’m just describing what I see in very careful, scientific language. And that’s OK!
There are strong arguments in favor of making psychology a descriptive science. Psychology research is still young (by comparison to physics or biology), and we may need some time to really get familiar with our subject matter. In many areas, the most interesting question still might be: “Are those two things related?” or “What happens when people are stressed, or are making decisions about money, or are deciding who to date?” Having an idea of what happens—even if we aren’t yet sure why it happens—is progress.
What I don’t see much value in is pretending that we’re making theoretical predictions. To me, a prediction that really comes from theory should be evident to anyone who is familiar with the theory. If a theory is capable of making a prediction about what should happen in a specific case, then everyone who has read the theory should make essentially the same prediction. There is no need for preregistration in that case, because it would be obvious when the write-up was twisted around to fit the existing data. If, on the other hand, you can see how the theory could be used to explain either the result reported or its opposite, then you should just ignore it. Until the theory can tell you what it really believes, it’s not ready for primetime.
So how do we get theories that are that specific? The most obvious answer is math. Stating a theory in the language of math—for example, as a formula or equation—makes it precise and unambiguous. We can also create formula simulations of what we think is going on and see if they (roughly) match the data we collected. We can also begin to create very specific verbal theories, but we need to make sure that we know how we can cash out each key term. When I say mood, do I mean specifically whatever someone checks when I ask them how they’re feeling on a scale from 1 (negative) to 10 (positive)? Or do I mean having a certain heart rate, blood pressure, and level of sweat on a person’s fingertips? If we are going to create theory with real stakes, we need to anchor our words to specific processes, and not assume that mood could mean any number of loosely related things.
As long as researchers are writing their manuscripts in the language of prediction—my theory predicted this result—we will need preregistration to keep ourselves honest. We shouldn’t go back to drawing bullseyes around our random shots. But as long as we need preregistration to prove what a theory really predicted, we won’t be dealing with very deep theories. The next step in the Credibility Revolution might just be a step away from theory, and towards description.