Verified by 
Can you handle the truth? Some of you may remember Jack Nicholson’s rant on the stand in A Few Good Men. I am often reminded of that phrase when I discuss research findings. Sadly, it is not because the findings are disturbing, though many are, but because as every good psychological scientist knows, research does not give you truths. This is a truth to keep in mind as you read headlines proclaiming “Research shows…”.
People like getting definite answers. We love it when science can tell us what to do. We sometimes fool ourselves into thinking that the latest finding on a topic (especially if it supports our view) is the truth about that topic. Eat this for health. Do not do that for happiness. When you are savvy about the scientific enterprise, you know that one study cannot be the end of the story. A scientific study only supports a hypothesis. One of the most frustrating two words I have to utter in discussing research is “It depends”. But that is the truth. Can you handle it?
Behavior is complex which is one major reason it is hard to get definitive answers. Results could change depending on how the research was conducted, which variables were measured, how the participants were recruited, how large the sample was, the context in which the study was done, and a whole host of other factors. Atul Gawande wrote about this persuasively in regard to medicine where he performed surgery on medicine itself showing it to be complicated, perplexing, and profoundly human, nothing close to the idealized “doctors have all the answers” world many believe it to be.
We also know that results of studies do not even replicate. You may be more likely to think of something as the truth if you get the same result every time. There have been many replication failures in science, again testifying to the complexity of research. Even if you repeated the methodology of a study completely, you may still not get the same results. But it gets worse.
If replication issues were not bad enough, there is now evidence that many scientists have not been entirely honest about how they did their research. In the most recent case, Dr. Brian Wansink resigned from Cornell University after a yearlong investigation showed he committed academic misconduct. In fact, in just the last month, six of his articles were retracted by the prestigious Journal of the American Medical Association (JAMA). A lot of his past studies on eating are now suspect. But it gets still worse.
Apparently even the very same data yields different results depending on who analyzes it. This week, Silberzahn and colleagues (Silberzahn et al. 2018) publishing an eye opening study. Twenty-nine teams involving 61 scientists tested one question: Are soccer referees more likely to give red cards to dark-skin-toned players than to light-skin-toned players? Here is the kicker. They were all given the same data set. The results varied. Twenty teams said YES. Nine teams said NO. Of these yay and nay-sayers, the strength of the effect was different.
WHY did this happen? The scientists used different analyses to test the questions. Different groups controlled for different factors. While some groups controlled for no factors, others controlled for up to seven different factors. But remember the data set was exactly the same.
This study is an astounding reminder of something rarely talked about. What the researcher does with the data makes the world of a difference. We are not talking fishing for results, fabricating numbers or worse, faking results. This study was an open window into how researchers are shaped by their habits. Researchers have pet analyses and set ways they tend to use when they approach research. The big reveal here is that these idiosyncrasies can make the difference in whether a finding is seen as true or false.
No research study gives you the truth. Each has many limitations. Make sure you treat the main findings with the caution required and do not get carried away with any one study. Science is self-correcting and inconsistency pushes us to do better science. And that’s good for us all.
