The famed physicist Richard Feynman once emphasized an underappreciated hallmark of science. He based it on an observation from the Second World War, when cargo airplanes dropped supplies on military bases in the South Seas islands. After the war, when the cargo planes stopped, the islanders invented a cargo cult, mimicking what they had observed, in the hopes of coaxing the planes to return. They created replicas of airplane landing strips; they put up lights; they signaled to the skies.
They did everything that normally preceded the landing of cargo airplanes. But the planes never landed.
Many pretend to practice science, Feynman argued, like the South Seas Islanders practiced air supply. They have all the bells and whistles; all the methods approximate standard science; but they lack something essential, something without which cargo cult science is merely a pseudoscience.
That missing piece is an attitude: Feynman called it a bending over backwards to see more than just one interpretation; an interest in refuting, and not just confirming, one's hypotheses; a wish not to fool oneself, first and foremost; a willingness to put one's own beliefs on the line - in short, "utter honesty" that goes far beyond everyday truthfulness. Feynman contrasted it with advertising: I might try to sell you my brand of soap, by talking up its strengths, minimizing its weaknesses, and ignoring its competitors. One need not lie to advertise in that manner; but neither is one being utterly honest.
Science as advertising is pseudoscience.
(Here is Feynman in his own words: "Details that could throw doubt on your interpretation must be given, if you know them. You must do the best you can--if you know anything at all wrong, or possibly wrong--to explain it. If you a theory, for example, and advertise it, or put it out, then
you must also put down all the facts that disagree with it, as well as those that agree with it...In summary, the idea is to try to give all of the information to help others to judge the value of your contribution; not just the information that leads to judgment in one particular direction or
another.")
We see this among those academics currently engaged in an anti-bipolar crusade. One blog post to which I responded consisted of an approach (ironic for critics of "corporate psychiatry") similar to advertising - providing one-sided data, ignoring contrary evidence, and silent on much that matters. Another knights templar group, about which I wrote my second blog posting ("Relax, you're not bipolar") about their initial article a year ago, is obsessed with the idea of bipolar overdiagnosis. Confirmed by CNN and Public Radio, this research group continues to publish in psychiatric journals with evidence that is non-evidence. Recently, another paper from the same study seeks to show that bipolar disorder is overdiagnosed when patients frequently have other conditions like borderline personality disorder or plain old vanilla depression (major depressive disorder, MDD). Yet the original paper on which that claim is based shows exactly the reverse: it proves that bipolar disorder is underdiagnosed, not overdiagnosed. How?
In the original study, about 30% of persons with bipolar disorder (based on the researchers' gold standard interview) had never been previously diagnosed with it (underdiagnosis); yet only 13% of persons had been diagnosed with bipolar disorder who did not really have it (overdiagnosis; again based on the researchers gold standard). Now, first grade mathematics teaches that 30 is greater than 13, and third grade teaches that it is over twice as great: thus bipolar disorder is over two times more underdiagnosed than overdiagnosed; thus it is predominantly underdiagnosed, not overdiagnosed.
There is another way of looking at the material, which makes the same point. Instead of the relative risk, this approach brings in the absolute difference. One effect could be 2% vs 1%; another could be 80% vs 40%; both are two fold relative risks, but the absolute effect of one is much larger than the other (80% is much larger than 2%). This measure of absolute difference - called the "Number needed to harm" (NNH) for bad outcomes - involves the reciprocal of the actual percentage difference. In this study the NNH for overdiagnosis would be the reciprocal of 13% = 7.7. The NNH for underdiagnosis would be the reciprocal of 30% or 1/0.3 = 3.3. (A NNT of less than 5 is considered huge, 5-10 moderate, 10-20 small. Usually this approach applies to drugs, so that one can say a drug with a NNT of 5 has major benefit. Here we are applying it to the likelihood of being underdiagnosed versus overdiagnosed). Using the data from this study, we can see that the chance of harm by underdiagnosis is huge, and by overdiagnosis, though not absent, is only moderate. Again, the data themselves show predominant underdiagnosis, not overdiagnosis.
How did the researchers ignore their own data? How could they come up with the opposite conclusion?
Let me bend over backwards.
Their conclusions follow, though still falsely, based on another way of looking at those data. I started by asking the question: What diagnosis did the patients have (confirmed by the researchers as bipolar, or confirmed not bipolar), and then what diagnoses had they previously received mistakenly? This is the way to show that someone was misdiagnosed, and how frequently (predominantly over or underdiagnosed).
The researchers' published paper does not report the above analyses. Instead they ask the question: Which patients were diagnosed as having bipolar disorder in the community? Then, how many of them were confirmed by the researchers to have bipolar disorder? The answer to that question was 42%; thus 58% were claimed to be overdiagnosed.
So, interpreted one way, the evidence shows that bipolar disorder is underdiagnosed; the other way, it seems overdiagnosed. How come the researchers only reported one interpretation?
An Epidemiology 101 course (which most psychiatrists never receive) would suffice to learn the difference between reliability and validity. Reliability means whether two clinicians, using whatever methods they choose, diagnose the same thing. If we agree to call X with the label "Y", then we are reliable. If the reality is that X is X, not Y, then our claim that X is Y, though reliable, is not valid.
In this kind of research, the question is whether clinicians are correctly diagnosing their patients. Their diagnoses are in question; they are not proven; they are being studied. Thus, to begin with clinicians' diagnoses is to ask the question of validity, not to answer it. We answer with the researchers' diagnoses in psychiatry, based on the standard research diagnostic interview that systematically assesses DSM-IV criteria. The analysis of clinicians' diagnoses is about reliability; it shows that we disagree on what we call bipolar disorder. It indicates that the bipolar diagnosis, in the community, is unreliably made.
It does not demonstrate overdiagnosis. Overdiagnosis reflects something different: it means that many people who do not have disease X are diagnosed with it, and predominantly so, meaning that not many have disease X and yet are not diagnosed with it. To answer that question, we really need to know who has the disease and who does not. We need to start with the proven diagnoses, the researchers' systematic interviews. That is the analysis I did; and it demonstrated predominant underdiagnosis.
