The Science Reform Effort and Its Discontents

Scientists embrace skepticism and reject accusations of shaming.

Posted Nov 17, 2016

A controversial essay by a former President of the Association for Psychological Science calling for civility in scientific discussions has inspired a great deal of commentary from scientific writers around the web.  You can read my synopsis of the essay here; you can read the whole essay here.

Because the status quo in psychological science includes problematic studies, practices, and findings, and because Fiske's essay seemed to be defending that status quo, many at the forefront of science reform took exception to it.  Why should you care?  Isn't the issue of quality and credibility of scientific research too technical for the intelligent layperson to understand?  First, not at all.  Most of the problems in psychological science are readily understandable to the layperson, and anyone suggesting otherwise is pulling a fast one (see link above).  Second, the credibility of anything you read about psychological science hinges on the quality of that science.  If that quality is suboptimal, you want to know it (at least if you are the type of person interested in psychology and you also prefer to believe things that are actually true, rather than some unknown mix of snake oil, hot air, bad practices, and self-serving self-promotion).

Many science writers viewed Fiske's essay as much more than a call for civility.  Fiske accused those engaging in criticism of psychological science outside of established peer review outlets as engaging in hostile and unethical actions.  As such, many have interpreted it as an attempt to defend the (what many believe to be dysfunctional) status quo in psychological science and shield it from justified criticisms.  

Here are some of what I believe to be the best, most trenchant comments about Fiske’s essay and science reform more generally:

TAMLER SOMMERS (VERYBADWIZARDS.COM, EPISODE 99, note that this is a podcast, and you can find the quote in the second segment, which begins about 20 minutes in):

“By ethical  she means the things that keep the status quo and the establishment in place …  From an outsider’s perspective it reads a little like, ‘You people outside the club don’t get to question our methods even if they are objectively questionable.  You have to go through us to do that, and we have to approve any criticisms you make of our methodology and our practices.”

ANDREW GELMAN (WHAT HAS HAPPENED DOWN HERE IS THE WINDS HAVE CHANGED).  Gelman's entire post is long but masterful and I encourage you to read the whole thing.  Here are some excerpts:

“The short story is that Cuddy, Norton, and Fiske made a bunch of data errors—which is too bad, but such things happen—and then when the errors were pointed out to them, they refused to reconsider anything. Their substantive theory is so open-ended that it can explain just about any result, any interaction in any direction. 

"And that’s why the authors’ claim that fixing the errors “does not change the

Source: Source: GetMillked, used with permission

conclusion of the paper” is both ridiculous and all too true. It’s ridiculous because one of the key claims is entirely based on a statistically significant p-value that is no longer there. But the claim is true because the real “conclusion of the paper” doesn’t depend on any of its details—all that matters is that there’s something, somewhere, that has p less than .05, because that’s enough to make publishable, promotable claims about “the pervasiveness and persistence of the elderly stereotype” or whatever else they want to publish that day.

When the authors protest that none of the errors really matter, it makes you realize that, in these projects, the data hardly matter at all.”

“… she’s working within a dead paradigm. A paradigm that should’ve been dead back in the 1960s when Meehl was writing on all this, but which in the wake of Simonsohn, Button et al., Nosek et al., is certainly dead today. It’s the paradigm of the open-ended theory, of publication in top journals and promotion in the popular and business press, based on “p less than .05” results obtained using abundant researcher degrees of freedom. It’s the paradigm of the theory that in the words of sociologist Jeremy Freese, is “more vampirical than empirical—unable to be killed by mere data.

“Fiske expresses concerns for the careers of her friends, careers that may have been damaged by public airing of their research mistakes. Just remember that, for each of these people, there may well be three other young researchers who were doing careful, serious work but then didn’t get picked for a plum job or promotion because it was too hard to compete with other candidates who did sloppy but flashy work that got published in Psych Science or PPNAS. It goes both ways.”

Source: Wikimedia Commons, "We are not amused."

“Fiske is annoyed with social media, and I can understand that. She’s sitting at the top of traditional media. She can publish an article in the APS Observer and get all this discussion without having to go through peer review; she has the power to approve articles for the prestigious Proceedings of the National Academy of Sciences; work by herself and har colleagues is featured in national newspapers, TV, radio, and even Ted talks, or so I’ve heard. Top-down media are Susan Fiske’s friend. Social media, though, she has no control over. That’s must be frustrating, and as a successful practioner of traditional media myself (yes, I too have published in scholarly journals), I too can get annoyed when newcomers circumvent the traditional channels of publication. People such as Fiske and myself spend our professional lives building up a small fortune of coin in the form of publications and citations, and it’s painful to see that devalued, or to think that there’s another sort of scrip in circulation that can buy things that our old-school money cannot.”

“Let me conclude with a key disagreement I have with Fiske. She prefers moderated forums where criticism is done in private. I prefer open discussion. Personally I am not a fan of Twitter, where the space limitation seems to encourge snappy, often adversarial exchanges. I like blogs, and blog comments, because we have enough space to fully explain ourselves and to give full references to what we are discussing."

And then there was this killer comment on Gelman’s post and Fiske’s essay by a person only identifying themselves as “Plucky” (who I contacted and received permission to quote in full):

PLUCKY’S BULLSEYE (Science is a method, not a community):

The key problem with Fiske is this sentence: “Ultimately, Science is a community, and we are all in it together.”

That is just flat-out wrong, and wrong in the ways that result in all that you have criticized. Science is not a community, it is a method.

I don’t know your field well, but I do know politics well. An overriding problem reading

 Clipart Panda
Source: Source: Clipart Panda

Fiske’s letter is that, unless one was told the context and background, it could easily be mistaken for the sort of weak, political defenses of interest groups of the op-eds-written-by-lobbyists variety. The verbiage, tone, responsible-adult-versus-unruly-child characterization of a dispute, unnamed strawmen bad-guys, gratuitous comparison to terrorism, and the false claim of lacking a personal stake in the fight are all standard issue rhetorical crutches of political hacks, and it ought to be embarrassing for a scientific field that someone who is apparently quite eminent in it would be talking this way. That’s really not a good look for Science.

"Obviously, Science is something practiced by people in collaboration, around which a community will naturally form, but the critical element to the authority and legitimacy of the enterprise is and has to be that all institutional arrangements be designed to counter-act the natural human tendencies of clique-ishness, careerism, etc and be continuously scrutinized for their efficacy in doing so. The entire point of granting tenure, for example, is precisely to shield the practice of science from the machinations of politics, and to allow people to do things like admit that they’ve wasted 10+ years chasing a phantom without losing their employment. Some concessions to human nature have to be made, but Science and its practitioners must always take the position that truth is more important than any scientist’s reputation or career.

"The fundamental problem with defining Science as “ultimately… a community” is that communities do not generally value the truth over their members’ well-being (in either the material or emotional sense). The functional purpose of a community is precisely to provide support, protection, and validation for its members. Communities whose stated purpose is altruistic need very strong commitments both individually and institutionally to avoid corruption, and even with those commitments the struggle to avoid corruption is eternal and never fully successful. Given the description of Dr. Fiske’s research interests she ought to know this better than just about anyone."

TAL YARKONI (THERE IS NO “TONE” PROBLEM IN PSYCHOLOGY)

"There is nothing wrong with the general tone of our discourse in psychology at the moment.

"Even if there was something wrong with the tone of our discourse, it would be deeply counterproductive to waste our time talking about it in vague general terms.
Fear of having one’s scientific findings torn apart by others is not unusual or pathological; it’s actually a completely normal–and healthy–feeling for a scientist.
Appeals to fairness are not worth taking seriously unless the argument is pitched at the level of the entire scientific community, rather than just the sub-community one happens to belong to."

"No researcher has a right to lead a successful career untroubled and unencumbered by any serious questioning of their findings. Nor do early-career researchers like Alec Beall, whose paper suggesting that fertile women are more likely to wear red shirts was severely criticized by Andrew Gelman and others. It is lamentably true that incisive public criticism may injure the reputation and job prospects of those whose work has been criticized. And it’s also true that this can be quite unfair, in the sense that there is generally no particular reason why these particular people should be criticized and suffer for it, while other people with very similar bodies of work go unscathed, and secure plum jobs or promotions."

But here’s the thing: what doesn’t seem fair at the level of one individual is often perfectly fair–or at least, unavoidable–at the level of an entirely community. As soon as one zooms out from any one individual, and instead surveys the field of psychology as a whole, it becomes clear that the job and reputation markets are, to a first approximation, a zero-sum game. As Gelman and many other people have noted, for every person who doesn’t get a job because their paper was criticized by a “replicator”, there could be three other candidates who didn’t get jobs because their much more methodologically rigorous work took too long to publish and/or couldn’t stack up in flashiness to the PR-grabbing work that did win the job lottery.

"...the idea that a finding published in one of our journals should be considered bias-free because it happened to come first, while a subsequent criticism or replication of that finding should be discounted because of personal motives or other biases is, frankly, delusional. Biases are everywhere; everyone has them. While this doesn’t mean that we should ignorethem, it does mean that we should either (a) call all biases out equally–which is generally impossible, or at the very least extremely impractical–or (b) accept that doing so is not productive, and that the best way to eliminate bias over the long term is to pit everyone’s biases against each other and let logical argument and empirical data decide who’s right."

NEUROSKEPTIC (ON “DESTRUCTO-CRITICISM”)

"Let’s suppose that Fiske is right and that some individuals, while pretending to be discussing science, are actually engaged in the targeted personal harassment of particular scientists. If that’s the case, what should we do?

"In my view, we should name names (or pseudonyms!): we should hold the offenders accountable with reference to specific examples of their attacks. After all, these people (Fiske says) are vicious bullies who are behaving in seriously unethical ways. If so, they deserve to be exposed.

"Yet Fiske doesn’t do this. She says, “I am not naming names because ad hominem smear tactics are already damaging our field.” But it’s not an ad hominem smear to point to a case of bullying or harassment and say ‘this is wrong’. On the contrary, that would be standing up for decency. If terrorists really are among us, we need to know who they are.

"Another reason why I think Fiske (and anyone else in a similar position) should name names is that it helps to draw boundaries. Fiske acknowledges that not all bloggers are bad: “Not all self-appointed critics behave unethically.” So who are the ethical ones? It would help to know some examples of the ‘good’ critics because we could then know where Fiske draws the boundary seperating good criticism from bad. As it stands, Fiske’s denouncations can easily be read as aimed at the vast majority of those who debate science online.

"In summary, I want to know who Fiske is calling a “destructo-critic” so I can judge the accuracy of the label. Am I one?"

NEUROCHAMBERS (METHODOLOGICAL TERRORISM AND OTHER MYTHS)

Wikimedia Commons
Source: Wikimedia Commons

"The truth is that we are in the midst of a power struggle, and it’s not between Fiske’s “destructo-critics” and their victims, but between reformers who are trying desperately to improve science and a vanguard of traditionalists who, every so often, look down from their thrones to throw a log in the road. As the body count of failed replications continues to climb, a new generation want a different kind of science and they want it now. They're demanding greater transparency in research practices. They want to be rewarded for doing high quality research regardless of the results. They want researchers to be accountable for the quality of the work they produce and for the choices they make as public professionals. It's all very sensible, constructive, and in the public interest." 

NEURONEUROTIC (THE DAY THE PALM HIT THE FACE)

I am sympathetic to the idea that criticism hurts, that we should perhaps be more mindful of just how snarky and frivolous we are with our criticism, and that there is a level of glee associated with how replication failures are publicized. But there is also a lot of glee with which high impact papers are being publicized and presented in TED talks. If you want the former to stop, perhaps we should also finally put an end to the bullshitting altogether.

URI SIMONSOHN (THREE IDEAS FOR CIVIL CRITICISM)

I think data-journalist is a more accurate metaphor than is data-police. Like journalists and unlike police officers, (academic) bloggers don’t have physical nor legal powers, they merely exercise free-speech sharing analyses and non-binding opinions that are valued by the people who choose to read them (in contrast, we are not free to ignore the police). Like journalists’, bloggers’ power hinges on being interesting, right, and persuasive.

Importantly, unlike journalists, most academic bloggers have similar training in the subject matter as the original authors whose work they discuss, and they inhabit their social and professional circles as well.  So bloggers are elite journalists: more qualified and better incentivized to be right

BENJAMIN KIRKUP (commenter on the Simplystatistics.org blog, Statistical Vitriol)

"In biomedicine, people are fighting for their lives and their immortality both. Those who are trying to keep a stable income through grants have their backs against the wall. Those who have published papers that they view as their legacy in the rolls and annals of science, have perhaps even more at stake.

James Lachlan Macleod
Source: James Lachlan Macleod

"In behavioral and social psychology, there is something else at stake too. Reading some of the statisticians, they talk about how all these unconscious biases must be assumed to be small effects, which is why the findings are notable and surprising and likely to be wrong (himmicanes, hurricanes, first class seating on the plane, manspreading, whatever). But this misses the whole point. The people who advance these theories believe earnestly that these biases are HUGE and constitute a new form of original sin that dictates individual human actions and organizational and institutional structures; that it must be measured, acknowledged, owned, accounted for, repented of, and paid for in mortification and penance. To say that the effects should be assumed to be small is basically blasphemy. Now, to attack the research as unfounded, particularly on the basis of Bayesian priors that refuse to see all our actions as grounded in this original sin - it is a conflict of basic world view, of religion. Of identity. It is a 'Haidt-style' conflict. As all know, attacking someone's livelihood, their legacy - and their religion?

"So the vitriol is in retrospect self-explanatory. A question is whether or how to disentangle statistics from it. That would require an exploration of alternatives, the question of science as a way of knowing, and so on. The stakes are obviously high."

BOTTOM LINES

1. I doubt the online presence of science reformers will be deterred by Fiske’s essay.

2. It is, in my view, good for psychological science that they won't be.

3. Fiske is undoubtedly right that some online discourse among scientists is incivil. Incivility is bad.  I completely agree that civil discourse should be encouraged and incivil discourse discouraged.

4. Incivil discourse among academics predates the new online media, and routinely occurs in the very outlets her essay advocated (e.g., peer review).

5. Many of the responses to Fiske’s essay show that problems and dysfunctions in the conduct of psychological science emerged from the old system, and are not likely to be solved without major reform of that system. 

6.  But, in the pantheon of problems plaguing psychological science, incivility is the least of its worries.  

Many believe scientific psychology is so riddled with problematic practices, that it has become difficult or impossible to know what is actually true and what is not.  It is the old system that produced this crisis.  The science reform effort does not have all the answers – but at least it is seeking them.

Attempts to suppress such criticism is deeply dysfunctional.