Social science research is in a sorry state. Just about anything can get published these days. Economists on fishing expeditions use fancy statistics to find correlations between soda and violence.
An article with a catchy title, "The Twinkie Defense: the relationship between carbonated non-diet soft drinks and violence perpetration among Boston high school students," came across my desktop this week. I thought I would review it here. The study collected survey data on over 1000 Boston high school students and found that teens that drink more soft drinks are more violent. There are many problems with this study and others like it but probably the major problem is that undergraduates and the public are likely to believe it.
Here is the first thing any psychology undergrad learns, "correlation does not imply causation." Just because the researchers found an association between drinking soda and self-reported violence does not mean that sodas cause violence. Next, because this is a survey and not a controlled experiment, several confounding variables could explain the association. For example, perhaps the teens that drink more sodas are hanging out with a gang in McDonald's and lack parental supervision. These two variables: peer influence and lack of supervision, may explain the violence. Also, the teens who reported drinking more sodas also smoked more cigarettes and drank more. The reported violence could be due to a cluster of variables like an impulsive personality, a frustrating interaction, and being hyped on caffeine. It is also known that with enough statistical manipulations to the data something can almost always be found. In grad school, we called this a fishing expedition. It was usually when faculty were under pressure to publish and their hypotheses failed. They fished for findings. That statistics can be used to mislead people has long been known. Mark Twain said, "There are lies, damn lies, and statistical lies."
But, the most troubling aspect of this study was how they treated the dependent variable, violence. There was no theoretical discussion of violence at all. What is violence? How is it measured? What does the previous psychological literature say about the construct? The authors simply asked teens if they were violent. They left it to teens to decide if they were violent. There is no operational definition of violence. Not only does this pull for social desirability, teens who carry weapons probably do want to appear violent. There was no external validity. Violence was not measured behaviorally, no peers or family were interviewed and no probation records were examined. So, what we are stuck with is the teens assessment of their own violence. Furthermore, this probably means different things to different people. To one, violence is a thought about hurting someone. To another, violence means an act, like murder. The survey did not discuss how it defined violence or explained the meaning to the subjects. There is no operational definition of violence.
Finally, the authors mention that the two potentially active ingredients in the sodas, sugar and caffeine, may explain the association but they don't even know which sodas the teens were drinking. Some of the sodas were caffeinated and some weren't. What kind of science is this?
This article is just one example of research that probably shouldn't be published. I don't mean to unduly focus on this one article. This is a universal trend. And with ease of internet publishing, the trend is growing. I understand the pressures on faculty to publish. This too, is just a reflection of two bigger social trends—universities as businesses and open source publishing. Experts, peer review, and universities as respected havens are dying. But, the problem with this trend toward publishing anything is that research loses its credibility. Most of the public can't evaluate social science research. And, the media is under deadline and space pressures to publish. These trends lead to publishing anything, even litter-ature.
