James C. Coyne Ph.D.

The Skeptical Sleuth


Study Did Not Show That Brief Therapy Kept Psychotic Patients Out of Hospital

Authors' claims ignore patients committing suicide or going to jail

Posted Aug 04, 2011

[In my last blog posting, I provided simple criteria that could be used to evaluate an article reporting a clinical trial evaluating a psychological intervention. I suggested that as an exercise, one particular report should be read carefully and that I would then provide my own assessment. If you have not done so, you might want to read the first blog before reading my evaluation. Or, you might want to check back to the earlier blog after reading this one.]

My Assessment: The authors were not justified in concluding that a brief Acceptance and Commitment Therapy (ACT) intervention reduced rehospitalization of psychotic patients by one half. The simple differences in rehospitalization between patients receiving the intervention and those in treatment as usual (TAU) were not significant. The "one half" statistic is misleading, particularly when patients who killed themselves or went to jail rather than being rehospitalized are taken into account.

Among flaws in the reporting of this trial are

  • We cannot confidently determine that rehospitalization was set as the primary outcome of the trial before data collection actually began.
  • At least 40 patients were assigned to each group, not the 35 on whom analyses were conducted. Patients were dropped from the analyses who had negative outcomes arguably as bad as rehospitalization-- suicide and going to jail. If a patient committed suicide, rather than getting rehospitalized, it can hardly be considered  a positive outcome.
  • We don't know whether the treatment as usual provided to the control patients was even adequate or whether the ACT patients got more of this treatment than the TAU patients.

Summary: Overall this is a small, flawed clinical trial that could be seen as actually having been a negative trial: a significant deterioration occurred in patients assigned to the ACT intervention in terms of the frequency of their symptoms. The abstract is not an accurate guide to actually happens in the study.

The abstract says "a rate of rehospitalization half that of TAU over a four-month follow-up", but the differences of 7/35  patients rehospitalized in the ACT group versus 14/35 patients in the TAU is not significant in a simple chi-square test. Moreover, it is noteworthy that there were 40 patients assigned to each group, not the 35 that were analyzed.

Was Rehospitalization Designated as the Primary Outcome Before The Trial Began? We cannot tell for certain, because there was no official registration of this trial ahead of time. But registration of psychotherapy trials was uncommon 2002, and still continues to be so today. In this regard, psychology has not caught up with medicine.

 Rehospitalization is described in the article as "the most objective outcome," but it is not one that is typically chosen to evaluate psychotherapy trials.

Significant differences are reported for "time to rehospitalization," but these analyses ignored patients who did not get rehospitalized, but who killed themselves or went to jail instead.

Psychotherapy trials presumably influence psychological variables like symptom severity or frequency. Influencing other outcomes depend on changes in these psychological variables being achieved.  

Rehospitalization is a complex social outcome that may only indirectly be influenced by psychotherapy. By analogy, a trial on which I was recently coinvestigator aimed to improve the adherence of HIV/AIDS patients to treatment. Our hypothesis was that the problem-solving intervention would improve adherence, which in turn would be reflected in reduced viral loads of these patients. We chose adherence as the primary outcome because it was behavioral, rather than viral load, which was a medical outcome that depended on adherence in order to evaluate the efficacy of the intervention.

The ACT trial assessed frequency of psychotic symptoms, which would seem to be a better choice for the primary outcome for a psychotherapy trial. But the patients who received the ACT intervention actually reported twice as many symptoms after getting the intervention, compared to patients who remained in TAU. This would be considered a negative trial under most circumstances. The authors offer no explanation except that the significant lower symptom reports of the TAU patients were a result of the TAU patients denying their symptoms. This is a novel interpretation that sounds like it was invented after the results were seen. Unfortunately, the measure of symptom frequency used had not been previously validated and so no comparisons possible with other trials or psychotic patients in other situations.

The trial additionally assessed how distressing the patients' hallucinations or delusions were and there were no differences between groups. Again, however, this measure was not previously validated and we don't really know what to make of it.

Finally, the trial assessed believability of symptoms, and ACT patients indicated that their symptoms were less believable. However this assessment was only made if the patients with symptomatic, and like the other symptom measures, it is not a validated measure use in other studies.

Reluctance to accept at face what the authors claim as the primary outcome is justified by the rarity of rehospitalization being selected the primary outcome in the existing literature. Moreover, symptoms would be more typically selected as the primary outcome but this study's results were negative for symptom frequency.

Authors often do not report or emphasize a primary outcome for which they did not obtain a significant effect, even though they should. It's not unreasonable for a reader to assume that the authors expected symptoms to be changed by the intervention. When they found that the change was in the wrong direction, they turned to  rehospitalization for the new primary outcome. Of course, we cannot know for sure because this trial was not registered beforehand.

What about the precise statistic of 'half the rate of hospitalization'? This statement in the abstract apparently refers to the 7 patients hospitalized who received ACT intervention versus the 14 who received TAU. However, even if we accept the authors use of a denominator of 35 patients per group, most patients in either group did not get hospitalized in the 120 days of follow-up. A simple chi-square statistic comparing 7/35 to 14/35 is not statistically significant. Think of it--- if only 1 of 50 ACT patients were hospitalized in 120 days versus three of 50 in the TAU, would we accept that ACT is three times as effective in keeping patients out of the hospital? This indicates the difficulty in accepting the summary statistic, even if it sounds like a powerful effect.

Are all the patients accounted for? Recall that the gold standard for reporting results of a clinical trial is intent-to-treat analyses, including all patients and analyses regardless of the extent to which they got exposed to a treatment or even whether they died from causes on related to the trial during the follow up period. The article reports "four participants in each condition moved out of the area, and one in in each condition died." The analyses should include these patients, but the question is how should they be included?

The conservative approach would be to count them as negative outcomes, rehospitalizations. So, we would count 12/40 among ACT patients versus 19/40 among TAU patients, a less impressive difference and still not significant.  But there is more...

The article is based on a dissertation, and the title of it seems suggest that symptoms, not rehospitalization, were the primary outcome: Acceptance and Commitment Therapy in the Treatment of Symptoms of Psychosis. I was able to retrieve a copy of the dissertation and I found that the article did not provide a complete and accurate account of what happened to these patients. The dissertation reports "One ACT and one control subject were incarcerated early in the follow-up, one in each group committed suicide, three ACT and two control subjects relocated, and one control subject did not follow-up with treatment to complete the outcome measures. Additionally, seven subjects consented to participate in the study, but were discharged from the inpatient unit before they could begin."

We do not know if the seven patients disappeared before their randomized. Regardless, the two patients locked up in jail and two committing suicide cannot be dropped from the study. And putting them back into the analyses reduces the analyses of days until hospitalization to meaninglessness.

Most patients in both the ACT and TAU group remained out of the hospital, a few went to jail or committed suicide, but among the patients who were rehospitalized, the ACT patients remained out of the hospital longer. But we cannot be more specific or readily conduct statistical tests, because we do not have a good way to take into account committing suicide or going to jail rather than getting rehospitalized.

Did the two groups differ and treatment received other than one group getting the ACT intervention and the other getting treatment as usual? This becomes somewhat of a moot question, given that the two groups did not differ in outcomes. However, a stay in a state hospital is usually quite brief and the care provided there is often inadequate. Although treatment may be available, there may be great variation in whether patients get much exposure to it. Unfortunately we are provided with no data concerning relative rates of other treatment of ACT patients versus TAU patients. It's quite possible that patients who have at least a minimal contact with the ACT intervention therapist also got encouragement and support from the therapist to take part in other available therapies on the unit or afterwards. We simply don't know.

Are there any other details relevant to evaluating this trial? Only a small minority of patients consented to participate in this trial. The authors estimate less than one in five could be recruited, and so hundreds of patients were approached. We cannot ascertain what bias there was in patients who were approached and agreed to take part in the study, but it does look like during a brief hospitalization, this treatment would not be accessed by most patients.

What is too good to be true probably isn't true. Hallucinating and delusional psychotic patients are unlikely to benefit from brief psychological interventions during acute hospitalization. Particularly when they are heavily medicated during their short stay, they may not even grasp the details of an intervention or remember them later. If we were to accept at face the claims of these authors, we would have to put aside reasonable expectations we have of such severely disturbed patients and the possibility of influencing them in a short time. There really isn't much of a precedence in the literature for such claims. Maybe ACT interventions are exceptionally powerful, or maybe these claims are exaggerated.

Broader Conclusion: We cannot always accept at face the conclusions of studies published in peer-reviewed journals, even prestigious ones. Stay tuned for more skeptical looks at highly cited articles, even those that have escaped any previous criticism.