Three Cheers for Methodological Terrorists
Psychology should embrace, not stigmatize, scientific criticism
Posted Oct 21, 2016
This is guest post by Terence Teo, a political scientist at Seton Hall University, is the third of a series on dysfunctions in (mostly social) psychological science. It was inspired by an inflammatory essay by the eminent social psychologist, Susan Fiske, which, in my view, reflects much of what ails social psychology. You can find the first two entries in this series here:
Now, On to Terence's Guest Blog:
I am an avid reader of Gelman's blog (on statistics and methods in psychology and beyond) and first came across Fiske's column there. That Fiske and her defenders resort essentially to ad hominem attacks on other scholars reveals a deeper, underlying issue with the field. I am no social psychologist, but it seems that what she is attempting to do is to rally the in-group of psychologists against the out-group of everybody else who, in her mind, is unfit to evaluate psychological research. In her view, the out-group uses "methods critique" to wage an insidious, unjustified war on innocent, unsuspecting psychologists, hence her use of the term "methodological terrorists."
Last I checked, all psychological research published in reputable journals contains "methods", and (statistical) methods is not simply the purview of psychologists, but all of the physical, natural, and social sciences. What's more, there is actually an academic discipline that focuses on methods, statistics. Is it only Ph.D. psychologists who have the training and therefore "right" to criticize psychological research?
Fiske hasn't identified who these innocent, unsuspecting psychologists, whose careers are being destroyed, are. Anecdotal evidence suggests that they are doing perfectly fine, bringing in tens of thousands of dollars per talk, and contributing to the infestation of TED talks (now synonymous with flashy research based on dubious evidence) on Youtube. She further calls these critics "self-appointed data police", which should be regarded as a term of praise rather than derision. Her piece brings to mind Irving Janis' groupthink, and in a rather scary way.
The central problem that Gelman and others raise is the stubborn refusal to recognize and acknowledge research design and methodological flaws when they are pointed out. I take the Popperian view that science is a provisional method of understanding, and that the central hallmark of science is falsification. The degree of confidence in what we know can never exceed the existing empirical evidence; more importantly, we must be humble enough to accept that we are wrong if the evidence runs contra to our theories. There are no facts, only knowledge given the evidence we have so far.
The replicability issue in psychology will be made a crisis by the refusal of the field to accept criticism from scholars outside the field. The big push in transparency, open
data and code, and reproducible research seems to be unable to penetrate the clientelistic relationships that permeate psychology (and medicine/biological sciences). Fiske's views in her column are part of this problem. The current system of grant funding is part of this problem. The current system of publishing is part of this problem. And this is harming the progress that psychology has made since the second world war.
Psychology is not in crisis because of incivility. It faces a looming crisis in part because of its refusal to openly welcome constructive criticism, its desire to be "popular" and "applicable to all", and its incentive structure. To be fair, I think some of the blame lies with the current state of methods training across the academy. We cannot all be statisticians, but we should, at the very least, listen and be willing to learn.
True peril lies in the resurgence of pre-WWII ideas, in the arguments that psychology needs to return to fiction and armchair theorists masquerading as philosophers with deep insight into the human condition based on anecdotes at best, and fables at worst. Today's (clinical) psychology has come far from the days of "it's your mommy's fault", false memories, multiple personalities, and Satanic rituals. Psychology cannot regress to those dark days; I suspect (hope) Fiske is concerned with the fallout from these critiques on the reputation of the discipline, and rightfully so. I just wish she had written something completely different.
(1) Floyd Allport counts as a methodological terrorist because of his devastating critique of the amazing!! world-changing!! New Look in Perception research, a critique that appeared in his book, Theories of Perception and the Concept of Structure
(2) Paul Meehl counts as a methodological terrorist because he routinely characterized psychological research as having nearly useless theories and empirical tests of those theories, an over-reliance on "cute" findings, and that it was often almost impossible to refute or disconfirm most psychological theories. His 1971 paper in The Journal of Social Issues was also probably the first to explicitly accuse social scientists of allowing their political preferences to distort their scientific conclusions