Verified by 
In the beginning there was google scholar. Google scholar was with god, and google scholar was god.
~ John 1:1 (paraphrased)
High school girls are those members of our species who are most concerned with popularity. Right? Wrong! The correct answer is: Scientists are. Only they call it reputation because they don’t want to sound like high school girls.
Scientists want impact, and impact means reputation and respect. Scientists may also want truth, heroically wrest it from Nature’s reluctant bosom. But they really want impact. Impact first and foremost. Truth is a nice side-benefit, proxy, icing on the cake. Impact pays. Department chairs, deans, and promotion committees reward impact, not truth. They can’t handle the truth (nor do they want to). We can’t know truth directly, says Popper, the Prophet (1959). You can know impact, though: numerically, inexorably, google scholarly. And the university administration knows it. A high citation count means a scientist has been noticed by other scientists. The work is being recognized, if not read. But a high impact rate is ambiguous. Perhaps many colleagues are impressed and inspired by the work, or they are appalled, or they simply cited the work because others did. It could be a citation cascade.
An international reputation is better than a domestic one. Does it mean that the findings on which this reputation is built are truer? Science itself cannot produce a correlation between citation counts (impact) and truth (truthiness, verisimilitude). For that, you need a psychic, a mystic, an oracle, or another prophet. If there were a study with this correlation as its result, what would ITS citation count be, and how would THAT reflect it ITS truth?
Mr. Stapel
In a Post-Popper, Post-Modern, Post-Stapel age, impact is sacred. Mess with impact, and you mess with god. Take Mr. Stapel, the defrocked Dutch deceiver. He made up the data for 50+ studies, published in all the scriptural places, and was found out. Oh golly, what an uproar! Sacrilege! A shanda! But did anyone scream and shout about the falsities Stapel propagated? No. The uproar was about the fact THAT he invented stuff, not WHAT that stuff was and how it might be related to REALITY. His “findings” actually sounded “plausible,” in a hindsight kinda way (Slovic & Fischhoff, 1977). Consider this morsel: Meat eaters, Stapel presumably found, are more selfish than vegetarians. Now who’s seriously suggesting that this is not a plausible hypothesis? It may be empirically false, but it is plausible by semantic triangulation alone. Meat eaters kill other sentient beings or have them killed. Vegetarians do not. Case closed.
As Stapel’s “studies” are being retracted, we regress to a state of ignorance with regard to the claims he made. We do not know if their opposites are true or if his claims might be true after all, in spite of him. This regression does not explain the moral outrage. The moral outrage in Stapel’s wake is about his exploitation of the impact-reputation system. Without a grip on truth, all we have is a social construction that pretends to be about truth, but is really only about the bundle of impact, influence, reputation, and respect.
If truth is absolute, reputation can only be relative (Anderson et al., 2012). We cannot live in a world where everyone is above average (the median, to be precise). A thousand citations on google scholar mean nothing if Median Joe comes in with two thousand. Stapel made everyone look bad because he had so many publications and citations. Now that he’s in forced retirement, others should feel enhanced. But they don’t. They feel violated because he exposed the quicksand on which their own reputations are built – except those, that is, who can point to a grain of truth found in a piece of research well done, and damn the citation count to hell (but who are they and what grain?).
Professor Gödel
Impact scores are pernicious because their credibility rests on the presumed relation between truth and impact. Saying that “If there is truth in a finding, then its impact will be great” sounds reasonable. We want it to be true, but we can have no independent verification. An independent verification of a statement regarding a relationship between truth and something else is logically impossible (I already claimed as much in paragraph 3) because such a statement would itself be a matter of truth vs. falsity. It’s a Gödel kind of thing. All we have is the reverse inference “If there is a high impact, then there must be a lot of truth.” As any beginning Bayesian knows, however, the validity of this reverse inference is unknowable if we don’t know the base rate of truth. And we don’t.
At least we know the base rate of high impact. 50% lies above the median – by definition.
Note. Gödel’s contribution, the incompleteness theorems, was a matter of mathematical proof, not data differing significantly from chance.
Anderson, C., Kraus, M. W., Galinsky, A. D., & Keltner, D. (2012). The local-ladder effect: Social status and subjective well-being. Psychological Science, 23, 764-771.
Popper, K. R. (1959). The logic of scientific discovery. London: Hutchinson.
Slovic, P., & Fischhoff, B. (1977). On the psychology of experimental surprises. Journal of Experimental Psychology: Human Perception and Performance, 3, 544-551.
