Three Ways to Criticize a Model
A criticism of how people criticize scientific models
Posted Nov 18, 2013
A model is a set of predictions that say “under x conditions, y should happen.” (It’s also a person who is photogenic, but if that’s what you’re interested in you’re not going to like the rest of this.)
Here’s the most common way to criticize a model: “That model can’t account for phenomenon ABC.” I wrote this post to point out a truism: this criticism can mean three very different things. To wit,
- The model makes a prediction about ABC and that prediction is wrong.
- The model makes a prediction about ABC and it’s right under some circumstances, but under other circumstances it’s wrong.
- The model does not make a prediction about ABC.
If you propose a model, and someone walks up to you at a conference and says “I have data that your model can’t predict!! Take that sucker!!” what do they mean? Often (or usually?) they mean criticism 2, or even 3. In my experience, it is common for people to treat these three types of criticism as more or less the same. They aren’t. There is a huge difference.
The first one is the most devastating. It says the model is wrong. The second says the model could be better—it is incomplete and sometimes wrong. The third isn’t much of an objection at all. Sure, making more predictions would be good, but let’s face it, no model is complete.
Here are some examples from cognitive psychology. As they show, the lines between the types of criticism are a little blurry, but I’d argue they are real.
My friend and colleague Shana Carpenter proposed a model in a paper from 2011. At this year’s psychonomics conference, another colleague, James Neely, pointed out a possible alternative explanation of Shana’s data. He tried to replicate her study with different materials. If he had gotten different results, it would have suggested that her theory might be wrong. He didn’t—her theory was supported—but his study was a test based on criticism 1. (Another classic example is the theory that vaccines cause autism. They don’t. This theory is just dead wrong. This criticism is most common when direct replications of a finding fail. )
- Bottom line: If criticism 1 is true, then the model is wrong.
Many studies show that taking test helps people figure out what they do and do not know. I recently found that this is true, but mainly when people take tests and do not check the answer afterward. If they do check the answer afterward it is still true, but the effect is much weaker. This is criticism 2. The original model is correct, but it is incomplete because it does not take into account feedback, which is an important variable because people usually do check the answer.
- Bottom line: If criticism 2 is true, then the model needs to be modified to make it more complete.
Here’s another example from this year’s psychonomics conference. I presented a model that assumes people forget information over time. That’s a pretty standard assumption, and the model makes predictions about something else (changes in the apparent rate of forgetting over time). A question was raised that I’ll convert into the form of a criticism: the model doesn’t account for reminiscence effects—times where memory increases over time, rather than decreasing. This is an important point, but even if it’s true it doesn’t invalidate the model—the model doesn’t make predictions about reminiscence that are different from any other conception of memory.
- Bottom line: If criticism 3 is true, then the model is probably fine. It just doesn’t cover everything under the sun. (Of course, it would always be better if it did.)
When a model doesn't predict a phenomenon (or dataset), the appropriate conclusion can range from “the model is wrong” to “the model is fine, but there are things it doesn't attempt to explain.” Don’t confuse one for the other.