License to Reject

How to mute and mangle scientific opinion.

Posted Jul 07, 2016

J. Krueger
Source: J. Krueger

Just the thing for unwinding after a rough day at the office. ~ Bond, James Bond

The Germans have this intriguing concept of Gesichertes Wissen, which loosely translates into “secure or secured knowledge,” that is, knowledge or at least ‘informed opinion’ that is scientifically obtained and validated. At one end of the spectrum, we have the conviction that reality is accessible to scientific observation and study, and that with due diligence we can get a portrait of the natural world. At the other end of the spectrum, we have the suspicion that all so-called knowledge is constructed, preliminary, observer-dependent, and prone to the tyranny of social power and manipulation. Most working scientist muddle through between these poles, leaning more to the former than to the latter. They agree, when pressed, that what is considered a scientific fact is, at the end of the day, a matter of sufficient consensus among so-called experts (see my essay on Ludwik Fleck’s and his concept of the Denkstil, which antedates Thomas Kuhn’s better-known concept of the paradigm). In that sense, there is an element of social power, control, and construction. The working scientist may not feel its force when doing the heavy lifting in the laboratory, but he (or she) feels the headwind of social ob- (er, con-) struction when submitting a paper (with objective findings) to be peer reviewed by a selective journal. Most papers are rejected, of those printed, many are not read, and of those read, many are not cited very much. It is hard, almost impossible, to have an impact on the field. Scientists who have not taken the deadwood route plug away, fueled by a sense of personal mission and overconfidence that they can prevail. Their plight, their myopic misunderstanding of the long odds of success, is an analogue to the market-entry dilemma faced by the would-be entrepreneur. “For many are called, but few are chosen” (Matthew 22:14).  

To get a paper published in a selective journal, i.e., to have any hope of making a recognized contribution, you need to convince the editor that you have a worthwhile story to tell. The editor in turn relies on reviewers for their opinion on the matter. The selection of the editor is a matter of sampling. Often you don’t know which associate editor will handle the paper. Then, the selection of the reviewers is also a matter of sampling, overseen by the editor and with little influence on your part. You may request certain individuals as reviewers (which always smacks of a conflict of interest) or ask that certain other individuals (which whom you have feuded) not be asked to write a review. At any rate, the vagaries of sampling, and a small sample it is, are heaped up and the outcome is, as Robyn Dawes once said, a “crapshoot.” In the collective, the body of published science is a pile of crapshoots that happened to be successful.

This is a dark view, as opposed to the sanguine perspective that the peer review system is the best we have, that it beats chance, and that in the long run we will get to know reality. In my day, I have written hundreds of reviews and I have read hundreds of reviews of my work. I have sat on editorial boards, and I have donned the hat of an associate editor for a selective journals. I have, as my grandfather Julius Welland, who saw action at the Western front, would have said “seen horses puke.” But today I write because a paper I submitted with a student of mine was mishandled egregiously. I have elected to share this episode with you so that you can take a look behind the curtain. This is a sample of one. It is not representative, but it is diagnostic. It shows a number of things that can happen, and it ain’t pretty. For delicacy’s sake, I have removed the names of the corresponding individuals, but I see no reason to withhold the name of the journal – well, OK, I’ll redact it.

The story of my Krexit

We submitted a paper on significance testing (see also this post) to the journal XXX YYY, noting that our paper was based on simulation and not mathematical derivation, and that psychologists would be the target audience. After a few days, the editor wrote:

Dear [authors]:

I write you in regards to manuscript [xxx] entitled "Go P!: Considering (Some) Critiques of Null Hypothesis Significance Testing" which you submitted to XXX YYY.

In view of the criticisms of the reviewers and editors, found at the bottom of this letter, your manuscript has been rejected for publication.

Thank you for considering XXX YYY for the publication of your research.  I hope this decision will not discourage you from submitting manuscripts in the future.

Yours sincerely,

He appended two statements from associate editors:

Associate Editor 1 Comments to the Author:
This paper is DEFINITELY NOT in the domain of mathematics. So, as an editor whose field is mathematics, I can only recommend rejection.

Associate Editor 2 Comments to the Author:
It seems to me on a quick read full of muddle and misunderstanding (e.g., I think they mean to use the Bayes factor in their point 1, not the probability of H given D (they aren’t being clear about their events), the point about 'probabilities of probabilities' is nonsense in my view, their comments about 'too many false positives' muddle true and false positives, etc.

That was all!

My reply was this:

Dear [editors],

No one can take your so-called review process seriously. Is this how you run your journal? Sending manuscripts to out-of-field reviewers and making decisions on the basis of vague one-liners? This is truly embarrassing indeed, we shall say.

To which the editor replied formulaically:

Dear [author],

Following submission of your manuscript, we sought expert advice from a number of Board Members on whether your paper should be sent for peer review and, regrettably, they recommended rejection on the bases described below. Whilst XXX YYY considers manuscripts from across the breadth of the sciences, we ask our Board Members to make an initial assessment of the scientific soundness of the paper – unfortunately, in this instance, the Board felt that your paper did not meet the quality threshold required of papers sent for peer review.

We certainly understand your disappointment in receiving this decision, but we wish you well if you choose to submit your paper elsewhere.

Thanks once again for considering XXX YYY for your research.

Best regards,

To which, in turn, I replied non-formulaically:

Dear [editor],

Thank you for responding but your explanation is not satisfying. The first associate editor recommended rejection because the ms was not mathematical and because he is a mathematician. The only inference one can draw from this is that this associate editor should not have received the paper. The second associate editor did precisely that which he accused us of doing, namely using muddled and imprecise language.

So no, this is not good and it is not professional. You have a responsibility to authors to treat them with professional courtesy and not like earthworms.

Good day.

This can happen, and as you see, it did happen. Manuscripts are being rejected without having been read (properly) or by individuals who feel that the papers ought to be rejected because they are not in their area of their expertise. The non-sequitur is mind-boggling. There is no proper system of oversight or appeal. Those one may appeal to are usually the same individuals (or their friends) as those who created the situation in the first place.

Let me close with one small substantive point, since it is the only one that emerges from this so-called correspondence. Reviewer 2 intones that the point about 'probabilities of probabilities' is nonsense. Well, this is rubbish, as the British might say. Suppose you have 2 urns, one filled with 9 red balls and 1 blue ball, and another urn filled with 5 red and 5 blue balls. The probability of drawing a red ball is .9 and in the former case and .5 in the latter. Now suppose you flip a coin first to decide which urn to sample. Heads you sample the p(red) = .9 urn, tails you sample the p(red) = .5 urn. And there you have it. It’s not rocket science, but you have to read the paper.

Kein Durchgang. I took the photo of the "Klausur: Kein Durchgang" sign at the Benedictine abbey of Neuburg near Heidelberg. A Klausur is a closed-off affair, originally among monks, now also among academics and the students they torture with exams. The main idea is that you can't get out once in. Here (Kein Durchgang) they are telling the rest of us to stay out. I did and went to the monasterial produce shop instead. They make a good Dunkel and Bernsteinweizen. This might be just the thing that Mr. Bond was referring to. Their goat cheese is not to be sneezed at either.