On Thin Ice: Is the Media Meltdown Justified by Evidence?
Regardless of political beliefs, biased research can mislead the public.
Posted Dec 20, 2020
Media bias is perhaps most obvious when discussing concepts that polarize individuals. A contentious category of dichotomous beliefs is the legitimacy of climate change. One typically believes in the long-term consequences of escalating global temperatures, or one does not. Subjective inferences are escalated when research designs are inadequate or erroneously scrutinized, yet the study is published in a scientific journal, or when research is lauded by writers who report research findings without methodological and statistical expertise. Misinformation perpetuates false beliefs among the public and is often cited as a cause of the “post-truth” era (Sinatra & Lombardi, 2019, p. 121), whereby individuals express views unsupported by confluent, replicable evidence but instead use personal experience or emotionally charged beliefs to substantiate their knowledge.
On Nov. 20, 2020, The New York Times (NYT) published an article authored by journalist Veronica Penney with the ominous title “Climate Change Is Making Winter Ice More Dangerous.” The article interpreted the work of biologist Sapna Sharma and her colleagues (2020), who used correlation data to investigate the relationship between drowning deaths and air temperatures across 10 different countries over variable time periods. According to the 16 researchers who conducted the original research study, ice-related drownings are increasing due to escalating freeze/thaw cycles that accelerate the probability of weaker ice and thus enhance the frequency of fatal drownings. On its face, this inference seems plausible, but under scrutiny the claim is indeed flawed and not justified by evidence.
Before outlining a series of methodological and interpretive concerns in the primary research study, it is important to understand why interpretation goes wrong, and the consequences of unjustified inferences. Chinn and Brewer (1993) in their seminal work on persuasion and conceptual change explained how individuals respond to contradictory scientific data that conflicts with personal understandings of the physical world. Their seven-step framework detailed reactions and responses to data considered anomalous. Within the framework, Chinn and Brewer explained that one approach to data inconsistent with beliefs is “reinterpretation,” which entails intellectual pondering and potential endorsement, but ultimately concluding that the new data is flawed, unclear, or irrelevant. When reinterpretation occurs, existing conceptions remain intact, confirming prior beliefs and justifying familiar inferences that are often communicated to others.
The Times narrative may be an example of reinterpretation with the noble motivation of warning the general population of the real-life consequences of catastrophic climate change. If you play on the ice you may drown, and the presumed cause of the fatality is hot air. While the journalist relies primarily upon direct quotations from the lead study author for her interpretations, the narrative, quoted below, perpetuates issues embedded in the original research:
1. “Drowning deaths are increasing exponentially.” This information is skewed by data from a limited number of higher incidence countries, while the authors intentionally excluded data from low incidence countries in the study analysis. Low sample sizes in some countries revealed that although “exponential” growth was described, actual death tolls there were as low as 10 per year, including many year-to-year fatality declines.
2. “Some of the sharpest increases were in areas where Indigenous customs and livelihood require extended time on ice.” Implying disparate impact on certain minority populations, the narrative fails to indicate precise exposure time, which likely explains more death variability than temperature according to the study authors. Some indigenous populations spend substantially more time on the ice as part of their work and livelihood and thus the probability of drowning increases as exposure increases.
3. “The coronavirus pandemic could also put more people at risk.” No data in the original study addressed causes beyond temperature fluctuation. Penney justifies this COVID inference by a statement from the lead study author who asserted in Ontario, “We have no place to go,” thus more time will purportedly be spent outside.
There is a common proclivity to imply causality based on correlation data. Considering the inability to control climate, unstandardized conditions across countries, variable reporting time periods, and dissimilar death categorizations by country, the methodology used in the original study is correlation at best.
What the study actually indicates
I now explain multiple methodological concerns with the Sharma et al. (2020) study.
1. The authors included 10 countries during data collection and excluded from the statistical analysis “Italy, Japan, and northern Canada from this analysis owing to the low number of drownings in these regions” (p.4). This exclusion appears to be unjustified; it is unclear whether including this data would negate the statistical significance of the reported findings.
2. The research design used by the authors is a variation of the statistical method known as linear regression. The regression process entails examining numerous factors to determine how much change in the outcome (i.e., drownings) can be attributed to factors in the statistical model (i.e., country, temperature). The authors revealed that 48% of the variance in drownings was accounted for by temperature. This means that an aggregate of other factors not measured in the current study (52%) contributed more to drowning deaths than temperature.
3. The statistical model used did not account for age, incidence of alcohol consumption, and type of vehicle if any involved (children, snowmobiles, and consumption are contributory to many ice drownings).
4. The authors reported they were “unable to acquire data on non-fatal drownings,” (p. 4) leading to the conclusion that their inferences were a “conservative estimate” (p. 4) of the influence of temperature on drownings. However, the exclusion of non-fatal drownings means the proportion of those on the ice who have drowned may have decreased, if the ratio of deaths to total incidents has declined. Absent knowing the precise frequency of ice activities, the inference of drownings increasing exponentially is decontextualized. Conventional wisdom would suggest that more time on the ice equals more drowning potential, yet the frequency on ice variable is unaccounted for in any of the analyses.
5. Country data covered different years and time periods. The aggregation of different time-period data means that prevailing conditions across countries are assumed to be similar. In reality, year-to-year conditions change.
This does not mean climate change is fake
In summary, although the evidence supporting climate change in most other studies is standardized and replicable, the embellishment in this case is unjustified by the data or the analysis. A common thinking flaw is confirmation bias, whereby the individual selectively chooses which data, in which form, and at which frequency to rationalize their pre-existing beliefs. It is unfortunate that the public may misconstrue these findings, when the concerns described here crystalize how the communication of research is often based on flawed premises and in this case some very thin ice.
Chinn, C. A., & Brewer, W. F. (1993). The role of anomalous data in knowledge acquisition: A theoretical framework and implications for science instruction. Review of Educational Research, 63, 1-49.
Penney, V. (2020, November 20). Climate change is making winter ice more dangerous. New York Times. https://www.nytimes.com/2020/11/20/climate/thin-ice-winter-drowning.html
Sharma S., Blagrave K., Watson S. R., O’Reilly C. M., Batt R., Magnuson J. J., et al. (2020) Increased winter drownings in ice-covered regions with warmer winters. PLoS One 15(11): e0241222. https://doi.org/10.1371/journal. pone.0241222
Sinatra, G. M., & Lombardi, D. (2019). Evaluating sources of scientific evidence and claims in the post-truth era may require reappraising plausibility judgments. Educational Psychologist, 55, 120-131.