On the Margin of the Impossible
Navigate through the flood of information on obesity.
Posted Nov 06, 2018
Harry, Lord Monchensey, has just returned to his family home after an absence of eight years to celebrate his elderly mother’s birthday in The Family Reunion, T.S. Eliot’s 1939 play in verse. Alas, he is veering toward insanity due to his own guilt: On an ocean voyage a year before, Harry’s wife has been “swept off the deck in the middle of a storm” and vanishes among the waves. “You would never imagine anyone could sink so quickly,” says Harry. He torments himself that he may have pushed her into the sea and been responsible for her death, “…the wish to get rid of her /Makes him believe he did…” His relatives, at a loss to help and concerned about Harry’s troubled mental state, consider inviting their local physician over for a consultation. Harry’s aunt Agatha, somewhat skeptical, says, “Not for the good it will do/But that nothing be left undone/On the margin of the impossible.”
It was Archibald L. Cochrane, an advocate for the use of randomized controlled trials and for whom the Cochrane Library database was named, who called attention to the Eliot quote. Writing about scientific research in general, in his now classic Effectiveness & Efficacy: Random Reflections on Health Services, (1971), Cochrane called for clinicians to “...abandon the pursuit of the ‘margin of the impossible’…” (p. 85)
The study of obesity borders on that “margin of the impossible” because, “Despite decades of research into the causes of the obesity pandemic, we seem to be no nearer to a solution now than when the rise in body weights was first chronicled decades ago.” (Hebert et al, Mayo Clinical Proceedings, 2013) “The circle of our understanding /is a very restricted area,” says the Chorus near the end of Eliot's play. How, though, can we make sense of where we are now?
Back in the mid-1950s, a Johns Hopkins researcher, studying the ease of abstracting information from his biological journals, wrote, “Perhaps no problem facing the individual scientist today is more defeating than the effort to cope with the flood of published scientific research, even within one’s own narrow specialty.” (Glass, Science, 1955) Tools were primitive and unsophisticated by 21st century standards: today we have capabilities of retrieval far beyond anything scientists then might have imagined, but the “effort to cope with the flood of scientific research, even within one’s own narrow specialty” has grown exponentially worse. By one account, as I had written eight years ago in my first blog, over 250 different professional journals, without even including, for example, journals in the fields of economics or consumer affairs, include articles relevant to obesity. (Baier et al, International Journal of Obesity, 2010) We are at risk of being inundated and like Harry’s wife, lost at sea, or rather, “lost in publication.” (Garg et al, Kidney International, 2006) What are some of the general issues involved in navigating this flood of information?
For one, “Not all scientific information is created equal.” (Ioannidis, PLOS Medicine 2018) For example, in their review of the current “medical misinformation mess,” Ioannidis and colleagues (European Journal of Clinical Investigation, 2017) found that there are about 17 million articles within the search engine of PubMed that involves humans, and apparently about 1 million articles are added each year. This is not particularly good news, though, since much of the information contained in these articles is misleading, unreliable, or of “uncertain reliability.” Furthermore, say Ioannidis et al (2017) most of those who read these studies are not even aware of this situation, and even if they are, most do not have sufficient proficiency necessary to evaluate the research studies they are reading.
Ioannidis (European Journal of Epidemiology, 2018) also called attention to the so-called Matthew Effect: Those papers that are heavily cited continue to be cited. Merton (Science, 1968) had described this effect, named for the Bible’s Book of Matthew (25.9): “For to everyone who has will more be given, and he will have abundance; but from him who has not, even what he has will be taken away." In other words, explains Merton, scientists “of considerable repute” keep getting greater recognition while those “who have not yet made their mark” have that recognition denied to them.
The media contribute to the problem, often by barraging the public with medical information, sometimes from those popular "authorities" on television who offer "evidence," much of which is “incomplete and wildly inaccurate.” (Ioannidis et al, 2017) . Since science is, after all, public, it must be communicated to others: “that is what we mean by a contribution to science—something given to the common fund of knowledge. In the end, then, science is a socially shared and socially validated body of knowledge.”(Merton, 1968) And it is “the best method we have of coming to an impartial knowledge of the world.” (Kroeger et al, American Journal of Clinical Nutrition, 2018) The media and even researchers themselves, though, ostensibly for well-meaning, even righteous aims, sometimes misrepresent or exaggerate, either consciously or unconsciously, scientific claims, i.e., what Cope and Allison have aptly labeled white hat bias. (Acta Paediatrica, 2010; International Journal of Obesity, 2010) (For more on white hat bias, see my blog 53) Clinicians, patients, and their families, as a result, are often left without the ability to evaluate treatment options.
One major impediment has been the persistence within the literature of thinking of obesity as a single disease with a single etiology. (Hebert et al, 2013; SR Karasu, American Journal of Lifestyle Medicine, 2013), though Stunkard and Wolff, as early as the 1950s, (Psychosomatic Medicine, 1958) noted that there was no need to presume a common etiology. Furthermore, instead of appreciating the enormous complexities of obesity, many researchers categorize obesity in the language of their own discipline. For example, physicians view obesity as a pathological state, i.e., a disease to be treated; sociologists may view it as an example of body diversity; the clergy, as an example of moral corruption and self-indulgence; anthropologists, as a disease of civilization; geneticists, as a genetic disorder; evolutionary biologists, as either appropriate or inappropriate adaptation to an obesogenic environment, with contributions from bacteria, viruses, endocrine-disrupting toxins, among others; physicists, as an energy imbalance following the laws of thermodynamics; and psychiatrists and psychologists, as a disorder of self-regulation or even addiction. (SR Karasu, 2013; SR Karasu, American Journal of Lifestyle Medicine, 2014.) (For more on the different “languages” see my blog 26, A Towering Babel.)
There are also methodological difficulties, some general to science and some specific to obesity studies. Particularly prevalent in obesity studies is that non-randomized observational research far outnumbers randomized controlled studies, and there is a careless use of causal language, particularly from these observational studies. (Trepanowski and Ioannidis, Advances in Nutrition, 2018)
Statistical errors are unusually common among obesity studies. “If you torture your data enough, they will tell you whatever you want to hear,” and “like other forms of torture, it leaves no incriminating marks when done skillfully…and may be difficult to prove even when there is incriminating evidence.” (Mills, NEJM, 1993) Allison and his colleagues (George et al, Obesity, 2016) identified 10 of the most common statistical errors seen in obesity research. One of the most common errors in obesity literature is assuming an intervention is effective when the study itself does not support that conclusion. (Brown et al, Proceedings of the National Academy of Sciences, 2018) Some other common errors include a mishandling of or even ignoring missing data or not dealing correctly with those subjects who don’t complete a study, ignoring confirmation bias, and ignoring regression to the mean. Confirmation bias is the tendency for researchers to evaluate their results differently or even somewhat less critically when their results match their initial expectations or conform to their initial hypotheses. Regression to the mean is a statistical phenomenon that occurs when repeated measurements are made on the same subject, and there is no control group to compare any difference from baseline. When the measurements change on repeated exam, (and often when subjects deviate less extremely from the mean) researchers can erroneously assume the change was due to their intervention. In other words, regression to the mean can “masquerade as a treatment effect.” (Kahathuduwa et al, Diabetes, Obesity and Metabolism, 2018)
Further, obesity research has been plagued by the complexities of inaccurate measurement, including those related to self-report of body weight, height, food intake, and exercise. “While one is either obese or not, the cutoff between the two states is arbitrary.” In other words, population health “manifests itself as a continuum…(and) “We can predict the health in populations with much more certainty than we can predict health in individuals.” (Galea,The Milbank Quarterly, 2018) “We call it health when we find no symptom/Of illness. Health is a relative term,” says the physician in Eliot’s play.
These measurement inaccuracies have led to what some researchers have called “pseudoscience.” (Trepanowski and Ioannidis, 2018; Archer et al, Current Problems in Cardiology, 2016; Archer et al, PLOS One, 2013) For example, attempts at nutritional surveillance, i.e., the systematic collection of data to detect trends in consumption and assess the connection between caloric intake and obesity rates over the past 40 years, have resulted in “pseudo-quantitative” data that are “physiologically implausible.” Data collected by the Centers for Disease Control from the NHANES population of civilian, non-institutionalized in the U.S. have relied on inaccurate and grossly misleading self-reports of food intake that have also excluded huge swaths of the U.S. population, including undocumented aliens, the homeless, and those institutionalized. Schoeller et al, in a letter signed by 17 leaders in the field of obesity research, documented how it has been over 20 years since Schoeller himself had found “substantial biases and inaccuracies,” i.e., “fatal flaws”—particularly gross under-reporting of caloric intake in obesity research. Unbelievably, the practice of self-report remains rampant in obesity studies. (Schoeller et al, American Journal of Clinical Nutrition, 2013; Dhurandhar et al, Journal of Nutrition, 2016)
Trust in nutritional science further diminishes when one study implicates a nutrient as harmful and then another labels the same nutrient as beneficial. Ioannidis names this extreme alternation the Proteus phenomenon, after the Greek god who could change his shape easily. (PLoS Medicine, 2005) Further, obesity research presents challenges because almost all nutritional variables are correlated with one another (Ioannidis, JAMA 2018): not only do we eat our carbohydrates, fats, and proteins in various combinations, but our foods expose us to thousands of chemicals, contaminants, and toxins that make it impossible to disentangle the potential influence of one component from others, as well as isolate environmental exposures and other variables such as lifestyle, education, socioeconomic status, etc. Further, adherence to a dietary protocol is often poor or the control group may adopt the experimental protocol. (Trepanowski and Ioannidis, 2018)
The public should be skeptical, writes Marion Nestle, in her book Unsavory Truth: How Food Companies Skew the Science of What We Eat (2018) whenever any study singles out any food, beverage, supplement, or specific ingredient that causes or reduces risk of obesity, heart disease, type 2 diabetes or cancer. (p. 228) Nestle calls the sensational findings on the benefits of single foods when they are removed from their “dietary context,” nutrifluff. (p. 54) Since we eat all foods in combination with others, it makes no sense to accept that one food has unusual and special benefits for our health.
In a particularly innovative study, Schoenfeld and Ioannidis (American Journal of Clinical Nutrition, 2013) raised the provocative question whether everything we eat is related to cancer. These researchers selected 50 common ingredients from random pages in a popular cookbook and found 40 of these ingredients (80 percent) were featured in articles that offered evidence for either an increased or decreased risk of cancer, despite weak statistical evidence. Gastrointestinal cancers, highlighted in 45 percent of the research, were the most commonly studied. Further, randomized controlled trials often failed repeatedly to find treatment effects for nutrients in which observational studies had previously reported strong associations, and even meta-analyses were sometimes biased and subject to misinterpretation. (Schoenfeld and Ioannidis, 2013) “If taken literally, if we increase or decrease intake of any of several nutrients by two servings a day, cancer will almost disappear worldwide.” (Brown et al, Advances in Nutrition, 2014)
Whatever its primary focus, nutrition research has been called “among the most contentious fields of science” (Ioannidis and Trepanowski, JAMA, 2018) because of the potential financial conflicts of interest from industry or other sources of funding, as well as researchers’ own potential biases and preferences (e.g. vegan, gluten-free, etc.) in what they eat or what causes they support. (Brown et al, 2014) Many researchers do believe it is a “puritanical and outdated view” that accepting funding from industry necessarily biases results. (Ioannidis, 2018) In fact, Allison et al found, in looking at top-tier medical journals, that randomized controlled trials were of equal quality regardless of the funding source. (Kaiser et al, International Journal of Obesity, 2012)
Nestle, though, offers a caveat, “Let me state for the record that financial ties with food companies are not necessarily corrupting; it is quite possible to do industry-funded research and retain independence and integrity. But food-company funding often does exert undue influence." (Nestle, 2018, p. 6) She adds, " (and)... it does suggest that the research question and interpretation require more than the usual level of scrutiny.” (p. 71) For Nestle, there should be a clear distinction between marketing by food companies and science. Further, Nestle sees financial conflicts of interest as categorically different from non-financial conflicts that can depend on individual beliefs, desires, and hypotheses that vary enormously from one investigator to another.
Though clearly not specific to those conducting nutrition studies, researchers have not been required to be transparent in releasing their raw data, with the result that many studies are unable to be replicated. Nestle (2018, p. 169) remembers the joke from years ago when she was a graduate student in molecular biology at Berkeley, “Never repeat an experiment that works on the first try.” In an effort to rectify the situation and preserve and make accessible these retrospective data, Hardwicke and Ioannidis (PLOS One, 2018) have launched an initiative—the Data Ark--an online repository for preserving raw data, encouraging scientific rigor, and increasing transparency among studies.
Just as Harry’s wife in T.S. Eliot’s play, is swept overboard and drowns, we are all drowning in a contaminated sea of publication. So much of the research within obesity yields pseudo-scientific data due to poor methodology, erroneous and unreliable measurement, and biases due to conflicts of interest. Ioannidis has suggested research itself needs its own study, what he and his colleagues have called meta-research as a way of verifying, evaluating, and rewarding research. (PLOS Biology, 2018) In science, there is sometimes a fine line between healthy skepticism and misrepresenting and exaggerating scientific uncertainty. (Allison et al, American Scientist, 2018) Though the "circle of our understanding" often seems "a very restricted area," researchers have no choice other than to navigate as best they can through the flood of information and away from that “margin of the impossible.”