Answering Mental Health Questions When Experiments Are Impossible
Science relies on experimentation, but what if that's out of the question?
Posted Apr 16, 2020
When we notice a change in the world, or in a lab, scientists wonder what caused the change. But how do we figure that out? It seems like a simple question, but there are numerous pitfalls along the way. Many scientists, including physicists, biologists, and psychologists, would tackle this question by performing an experiment. By manipulating the specific conditions of different groups that are otherwise the same, scientists can determine cause for a variety of effects. A biologist might grow the same bacteria in two dishes and squirt a new antibiotic on one and water on the other to see if the outcomes are different. By keeping everything else exactly the same, if they see a change in one dish but not the other, they can be confident that the new antibiotic works.
However, there are many differences in the world that we might be interested in explaining, but not all of them can be examined in a controlled way, like with dishes of bacteria in a lab. Designing a good experiment isn’t always easy, and there are particular challenges for anyone wanting to design experiments in epidemiology. When a drug is developed (and this could be a vaccine for an infectious disease or an antidepressant, as an example of a non-infectious, chronic disorder), large-scale clinical trials are required to determine if a drug works. These are giant, high-stakes experiments, and they are crucial for helping us find cures to pernicious afflictions.
Unfortunately, clinical drug trials aren’t usually designed to tell us much about mechanism—that is, the underlying cause of an illness—particularly with psychological disorders. In attempts to get at answers, many research psychologists in this area use large, longitudinal samples of volunteers to try to determine the cause of disorders later in life by looking back for risk factors earlier in life.
The problem with this approach is that these data tend not to include any experiments. The data are just records of how different people have lived their lives, and we try to retrospectively identify explanations for what happens to people. This introduces an age-old issue, the problem of correlation versus causation.
Let’s say that children who are socially anxious also tend to have fewer friends, which there is evidence for, according to research published in Human Psychiatry and Child Development. It would be easy to assume that these kids’ anxiety causes them to form fewer friendships, but we cannot draw this conclusion from the evidence. It could be that having fewer friends causes more social anxiety, or both could be due to a third factor we’ve not considered, such as that more talkative children tend to have more friends, and they also tend to have more positive emotions, which is related to lower social anxiety. In these situations, it is far from clear what causes what. Instead, these factors are merely correlated: if a child is socially anxious they will also tend to have fewer friends, and vice versa, but that’s it.
The fact that these observational data often do not let us answer those “what caused this” questions is a major issue, particularly when some media outlets are more keen than scientists are to make claims about causation when they are not warranted. Nevertheless, research of this sort is not hopeless, and one thing this blog is about is how researchers are able to make informed deductions about causation without experimental data.
Take depression and cognitive function. In older people, cognitive function tends to decline, and depression symptoms tend to increase, so the two are correlated, but does one cause the other? Depression has long been considered a risk factor for cognitive decline and dementia. However, some recent work in the journal Psychological Science demonstrated strong evidence to the contrary, finding that lower cognitive function is linked to later increases in depressive symptoms. This study used an advanced design enabled by the sample under study: the Lothian Birth Cohort of 1936. In this cohort, cognitive function and depressive symptoms were collected every 3 years and everyone was the same age, so there are good measures of cognition and depression at ages 70, 73, 76, and 79. The authors then used these data to look at changes in cognition and depression: whether cognitive scores or depressive symptoms earlier in the eighth decade are related to changes later in the eighth decade in the other variable.
For example, between ages 70 and 73, we might see individuals with higher cognitive function experience a decrease in depressive symptoms, while individuals with lower cognitive function experience an increase in depression. This is, in fact, what the authors found, and it suggests that in this group of older people, cognitive dysfunction is linked to later depression, which is the opposite of prevailing views on depression and cognitive dysfunction. This is an example of reverse causation (hey, that’s the name of the blog)—when the apparent direction of effect, and possibly causality, goes against the common presumption.
There are caveats to this example, too! It is just one group of people, from a particular place, followed at a particular time. Moreover, the data are still not experimental—we can’t forcibly change someone’s cognitive functions and see if they become more or less depressed. This study also does not tell us much about mechanism. Maybe cognitive function is related to later changes in depressive symptoms, but if so, why and how? That’s still an open question.
Working with samples and data from big surveys is challenging, both because the data are not experimental, and because these are big data sets, which make them very powerful for finding links. Unfortunately, links are often all too easy to find, and it is the responsibility of us researchers to do our best to identify if there is a causative link, or merely a correlation.
Aichele, S., Ghisletta, P., Corley, J., Pattie, A., Taylor, A. M., Starr, J. M., & Deary, I. J. (2018). Fluid intelligence predicts change in depressive symptoms in later life: The Lothian birth cohort 1936. Psychological science, 29(12), 1984-1995.
Hughes, A. A., & Kendall, P. C. (2009). Psychometric properties of the Positive and Negative Affect Scale for Children (PANAS-C) in children with anxiety disorders. Child Psychiatry and Human Development, 40(3), 343-352.
Jorm, A. F. (2000). Is depression a risk factor for dementia or cognitive decline?. Gerontology, 46(4), 219-227.
Newcomb, A. F., & Bagwell, C. L. (1995). Children's friendship relations: A meta-analytic review. Psychological bulletin, 117(2), 306.
Scharfstein, L., Alfano, C., Beidel, D., & Wong, N. (2011). Children with generalized anxiety disorder do not have peer problems, just fewer friends. Child Psychiatry & Human Development, 42(6), 712-723.