They've become hard to spot because we're so used to them. But they're there. And they're worrisome. Here are two problems with the way we do research on education and training.
Problem 1: Double blind
Yesterday, Boot, Simons, Stothart, and Stutts (2013) published a paper entitled The Pervasive Problem With Placebos in Psychology (or see Ed Yong's nice summary here). In psychology studies, people usually know what condition they are in. If one group plays video games and one does not, the video game group knows it. The problem is, they might expect to do better on tests of cognitive ability. If they actually do better, you have to wonder: was it the video games or the expectations?
Expectations matter. That's why there are placebos--so everyone expects to get better. In fact, larger placebo pills have larger effects than smaller ones, because people expect them to be more effective. And sham surgery (yes, that means doing cutting someone open, doing nothing, and then sewing them back up) has an even larger effect than pills, because people expect more from it.
Proper medical testing makes sure the subjects are blind to what treatment group they are in. Everyone takes a pill and no one knows whether their pill contains medication or not. In a double blind study, the subjects and the researchers are both blind to who is in what condition, because researcher expectations can also affect subjects.
It can be very difficult to run double blind studies in research on training and education, because unlike a pill, one kind of training does not look like another. Subjects know what treatment they're getting. This is a fundamental problem that's hard, sometimes impossible, to avoid. Boot et al. (2013) make usefull suggestions, though; the foremost is, measure expectations if you can't avoid them!
Problem 2: Realistic comparison
Would you take a new drug if it was better than a placebo? If you were a doctor, would you recommend it to your patients? It depends on a crucial question: Is the drug better than the current standard practice? If there's already something better, of course it's not the best choice. See Figure 1.
In a lot of psychological and educational research, we don't ask this crucial question. Often, we compare a new treatment to a control condition that is neither effective nor standard--basically, it's a placebo--and then, when the treatment is better, we recommend it (see Figure 2).
This is the second problem with control conditions in training and education. It is summarized in a recent paper by Kornell, Rabello, and Klein (2012) and it is actually easier to solve than the first, which makes it, in some ways, more embarrassing.
Let me elaborate by picking on my own research: I've often recommended people do more testing in school and at home, because it's a better way to learn than re-reading. But teachers almost never ask students to simply re-read something they've read before--which is the control condition in the vast majority of experiments on testing effects.
Moreover, we know that re-reading isn't very effective. It would make more sense to recommend more testing if we could first demonstrate that testing is better than the activities that teachers are actually implementing on a daily basis. Is testing better than holding a class debate? Or informally asking students questions? Or techniques like elaborative interrogation?
In medicine, we wouldn't accept a new treatment unless we knew the answers to questions like these. A study that had a big fat question mark in place of the current treatment, like in Figure 2, would not cut the mustard. Why should training and education be any different?
Find me on Twitter.