Researchers Promote Questionable Math App in Major Journal

One of the top journals in psychology released an article today that I found misleading.

Schaeffer, M. W., Rozek, C. S., Berkowitz, T., Levine, S. C., & Beilock, S. L. (2018). Disassociating the relation between parents’ math anxiety and children’s math achievement: Long-term effects of a math app intervention. Journal of Experimental Psychology: General, 1(999), 1–9. http://doi.org/10.1037/xge0000490

The study examined a diverse sample of children from the Chicago area. In first grade, these kids were given an app that involved working on schoolwork with a parent. There were two versions of the app. One was all about math. The other (which served as a control condition) was about reading. The question was whether the app would improve math performance. The researchers followed up with the kids and examined their math skills in third grade. Now the results are out.

Did the app make kids better at math? No.

Overall math performance was not significantly different between kids who had the math app and kids who had the reading app. As the authors write "...it should be noted that we do not see a main effect of the intervention."

To be clear, the app was helpful: Students in the control condition did worse in math if their parents had high math anxiety (which seems bad). In the math condition, this relationship wasn't significant (which seems good). The authors summarize their findings as follows:

"We show that a particular math app can lead to sustained changes in the association between parents’ math anxiety and children’s math achievement, and in future research, we hope to identify the specific factors that make this app successful."

This is true. And it seems potentially important. But I also think the article is misleading. In this example, you have to remember not to assume "successful" means "makes kids learn more math." Worse, the abstract refers to "the sustained benefit of the math app." It also says that interventions like the math app "...can have powerful lasting effects on children’s academic achievement..."

Articles should be as open and up-front about their findings as possible. This one is hiding something important. It is a barrage of positivity. In a two-page results section, a single sentence points out that the intervention didn't have a significant overall effect. It's in an inconspicuous position, in the middle of a paragraph in the middle of the results section. The rest of the results section is entirely upbeat, as is the rest of the article.

What really gets me is how the authors imply that this app is a big gift to society. For example, the authors write:

"Although many apps are advertised as being able to develop children’s academic skills and abilities, there is a dearth of evidence about the effectiveness of these apps and their impact on children’s learning and even fewer studies that look at the lasting effects on learning. Additionally, all apps are not created equal (Hirsh-Pasek et al., 2015), and the math app intervention examined in this study differs from many traditional apps in several potentially significant ways."

Ironically, the authors criticize the practice of making misleading claims about "lasting effects on learning," but that is exactly what they are doing. Unfortunately, it's nothing new. As I've written before, educational psychologists frequently promise more than their results deliver.

Here's the wording I would have liked to see: "Overall, the math app did not improve math scores significantly more than the control condition." That's the truth. If was in the abstract and the conclusion section, there would be no problem with this article.

From the authors' perspective, though, there might have been a problem. It might have decreased the chance that this paper would have been published in such a prestigious journal. It might also make the app seem less appealing.

If someone tried to sell me this app, I wouldn't buy it. My first question would be: "What evidence is there that this app improves math skill overall?" Based on the 3rd grade data in this paper, there isn't any.

In short, this paper promises more than it can deliver. I'm not saying the authors did this on purpose. They probably have good intentions. I am sure they are eager to help kids, and I wouldn't blame them for being over-eager. But their intentions are not for me to judge.

What I can say is, the writing reminds me of the careful wording of a politician. It seems to say something important, but when you parse the language carefully, you realize something at the core is missing.

Anxiety Essential Reads

How to Grow Your Tolerance for Uncertainty

How Long Do Panic Attacks Last?

Check me out on Twitter.