Carl Sagan is credited with popularizing the dictum “Extraordinary claims require extraordinary evidence” during an episode of his groundbreaking TV series Cosmos. While it is a nice dictum in principle, it does not work in practice.
The dictum has recently been revived and is currently in frequent use, after the Cornell psychologist Daryl J. Bem published his paper “Feeling the Future: Experimental Evidence for Anomalous Retroactive Influences on Cognition and Affect” in the Journal of Personality and Social Psychology. The paper presents evidence, collected from nine separate experiments, for precognition, an ability to sense the future state of the world before it happens. The paper went through the same process of rigorous peer review as any other submission to JPSP, which is one of the leading journals in social and personality psychology, and was published because it satisfied the reviewers and the editors of the Journal. Yet Bem’s critics claim that his paper should have been held to a higher standard of evidence because his claim that humans possess extrasensory perception (ESP) is extraordinary. The critics invoke Sagan’s dictum “Extraordinary claims require extraordinary evidence” in their own claim that Bem’s paper should not have been published.
The problem with the dictum is that there are no absolute criteria for what counts as “extraordinary claims.”
In particular, what counts as extraordinary depends entirely
on what you know and believe.
In the extreme case, if you know nothing, then everything
is an extraordinary claim.
As the comedian Elayne Boosler used to quip, “Popcorn is magic if you don’t know how it happens.”
Even less extremely, whether a given scientific claim counts as extraordinary depends entirely on what you know and believe. The same claim can be extraordinary for some but not for others. I personally don’t believe that, given what we know from quantum mechanics, Bem’s claim of precognition – that human cognition and affect can be influenced by future events – is particularly extraordinary. What is extraordinary is quantum mechanics, not precognition, yet quantum mechanics is a well-accepted (if poorly comprehended) part of modern physics.
Worse, what counts as extraordinary depends also on the scientific fads and fashion of the time. The claims of race and sex differences in intelligence were not at all extraordinary a hundred years ago. They are considered to be extremely extraordinary today, requiring extraordinary evidence. They will no longer be extraordinary in a hundred (or, hopefully, five) years from now.
The biggest problem with the dictum “Extraordinary claims require extraordinary evidence” is that it is often and usually used politically. It is used too often to silence and censor politically incorrect claims that people do not like. “Extraordinary claims require extraordinary evidence” often transforms into “Politically incorrect claims require extraordinary (and impossible) evidence.” As Robert Kurzban notes, this is the major reason evolutionary psychology is often held to higher standards of evidence.
It is perfectly fine to hold evolutionary psychology (or any other scientific field) to higher standards of evidence, as long as all other fields are held to identical standards.
The required level of evidentiary standards cannot depend on the conclusion or the contents of the claims, especially, on how politically correct or popular they are.
Until or unless we can derive an absolutely objective definition of what counts as an extraordinary claim, science will fare better if we all forget about Sagan’s dictum and hold all scientific claims to identically high standards of evidence.