The news item continued:

• Neither antidepressants nor "talk therapy" were able to outperform inactive placebo pills in a new clinical trial on depression treatment -- though there were hints that the effects varied based on people's sex and race, researchers report.
• After 16 weeks, there were no overall differences in how the three groups fared.
• Of antidepressant patients, 31 percent were treatment "responders" (meaning they'd fallen below a certain score on a standard measure of depression symptoms, or had seen their score drop at least 50 percent.)
• The same was true of about 28 percent of patients in the talk-therapy group, and 24 percent in the placebo group. The differences among the three groups were so small as to be likely due to chance.
• "I was surprised by the results. They weren't what I'd expected," said lead researcher Jacques P. Barber, dean of the Institute of Advanced Psychological Studies at Adelphi University in Garden City, New York.

Immediate Grounds for Skepticism.

• This finding is contradicted by a large literature demonstrating that antidepressants are superior to pill placebo and a smaller literature showing that psychotherapy has a similar advantage when measured against a pill placebo provided in a clinical trial.
• Supportive expressive therapy, a short-term version of psychodynamic and psychoanalytic treatments, lacks empirical support for depression, particularly compared to the well-validated cognitive behavior therapy,. So, the label "talk therapy" in the news item is too broad and inaccurate.
• The rates of response to both antidepressants and pill placebo in this trial are lower than in other trials.
• This trial is too small to detect differences among active treatments, even if they were present.

Going to the Original Article.

I was most interested in whether this trial succeeded in obtaining the intended sample size, if the trial was analyzed with "gold standard" intent-to-treat analyses that take into account all patients who were randomized, and if sufficient numbers of patients obtained adequate exposure to treatment.

The article acknowledges difficulties recruiting the intended sample size of 180 patients. A number of different strategies were adopted to recruit 156 patients, including ads in newspapers. The article does not state what incentives were offered, but such recruitment strategies require financial incentives to attract and retain patients Substantial payments have the disadvantage of attracting patients who are motivated by the money, not improvement in their depression, . Many "professional research participants" earn a good proportion of their income from volunteering for clinical trials.

For patients with insurance, treatment is already available in the community from primary care physicians. The antidepressants used in the study have already gone generic, cheap versions are available for a few dollars a month. Additionally, patients with insurance have little incentive to enroll in a clinical trial where they might not get their preferred treatment. These considerations become important in a trial where there are no innovative treatments: a rather traditional psychodynamic treatment versus a cheap, generic antidepressant.

So, this is an unrepresentative sample that likely lacks the motivation of depressed patients seeking treatment in clinical settings. The sample was low income--typical of Philadelphians enrolling in clinical trials with financial incentives. But the article indicates no special steps to educate patients about treatment, increase their adherence, or retain them in the study. That missing ingredient may have contributed to the 40% dropout rate. My group found that it takes persistent effort, special accommodations like flexible scheduling of contacts, and incentives to recruit and retain representative samples of low income, urban research participants.

The study is flawed by too many patients getting an inadequate exposure to treatment. Only 91 of the 156 patients completed the study. The investigators attempted to compensate for the dropouts by using a statistical technique known as Last Observation Brought Forward (LOBF). This technique considers the last outcome data collected from a patient as that patient's final outcome. LOBF is known to provide biased estimates of group differences in the outcome of a clinical trial. It assumes dropouts are random and ignores whether patients were improving or deteriorating when they dropped out.

Thus, use of LOBF allowed the investigators to have data for all patients, even dropouts. Including all patients who were randomized--- what are called intent-to-treat analyses-- is the gold standard for clinical trials. Such analyses answer the question of what happens if patients are randomized to a particular treatment. If many patients drop out, that is a relevant outcome. Moreover, limiting analyses only to patients who completed a trial, no longer preserves the benefits of randomization. Dropouts are not random. So, the best strategy is to rely on intent-to-treat analyses, but relying on the LOBF introduced bias in this trial. Still, it is also important to pay attention to "as treated" analyses focusing on patients who actually got the treatment as planned. And here we see these numbers get a lot smaller.

The odds were against finding any differences between treatments. According to the statistical power analyses used to design the study, if the investigators had gotten the sample size they planned, there would have been an 80% chance of finding a difference between either the psychodynamic therapy or the antidepressants and an inert treatment, such as waitlist control. However, the problem is that pill placebo administered in the context of a clinical trial is not an inert treatment, as the investigators note. Both patients and providers are blinded, so they do not know the patients getting a placebo are not getting an antidepressant. The patients are given positive expectations and a lot of encouragement and support that may be sufficient to produce improvement.

So, the initial power calculations used to determine sample size were unrealistic because they assumed that the comparison was with an inert treatment. The intended sample size of 180 distributed across three groups is too small. But we should take into account that the investigators succeeded in recruiting only 156 patients, with 40% dropouts among patients getting medication or pill placebo and 23% of the patients getting psychotherapy, the chances of obtaining a significant effect are well below 50-50, especially if any effect depended on patients actually having been sufficiently exposed to treatments. The investigators had little chance of finding a statistically significant difference among the three groups. Why was the lead investigator surprised by the results?

In the larger literature, getting an antidepressant versus a pill placebo in a clinical trial has a small effect size (about r = .30). In the entire literature, there are less than a dozen comparisons between psychotherapy and pill placebo, but the advantage of psychotherapy over placebo is about the same as for an antidepressant. This, of course, is an average effect, with some patients doing much better and others worse.

This does not mean that a primary care physician prescribing a sugar pill to patients would have a similar sized positive effect. Actually, given the poor quality of routine care for depression in the community, antidepressants given by a primary care physician have about the same effects on depressed patients as pill placebos given in the context of a clinical trial, where there is support and active clinical management and, importantly, follow-up.