As regular readers of the New York Times and other major newspapers are aware, Wikipedia recently triggered a scandal in the psychological community by posting the much cherished Rorschach inkblots - first published by Hermann Rorschach in 1921 - online. In fact, this was not the first public "outing" of the blots (or at least the outlines of the blots), but the New York Times' decision to accord Wikipedia's unveiling of the blots front page coverage on July 29th generated a major internecine battle between proponents and opponents of the test. Not surprisingly, most devotees of the test were appalled. One visitor to a Website wrote: "It's disgraceful! Wikimedia should pull down or obscure those images!...Publishing these Rorshach (sic) images can have the same devastating (sic) effect to a potential patient, a sick individual who needs help, and could be harmed because he got access to the test material via Wikipedia." In contrast, most critics of the test were blasé, to put it mildly. One yawned that he hoped the Rorschach would eventually be found along horoscopes in daily newspapers; another remarked wryly that the Rorschach should long ago have been burned along with witches. To a substantial extent, responses to the Wikipedia entry paralleled people's initial attitudes toward the test, rendering the public and scientific reaction to the posting something of a Rorschach in and of itself. By the way, if you're curious, you can find the now infamous Wikipedia entry, along with thumbnails of the actual blots, here. http://en.wikipedia.org/wiki/Rorschach_test
Was making the inkblots public a good idea? I'm torn. On the one hand, most research on the Rorschach indicates that with the exception of a handful of clinical purposes, like detecting thought disorder (as in schizophrenia and bipolar disorder), measuring interpersonal dependency, and perhaps predicting the prognosis of psychotherapy, the Rorschach isn't particularly valid or useful as a clinical instrument. Even most of its advocates acknowledge that it's not especially helpful as a diagnostic test, and most studies show that it correlates weakly with personality traits of interest to clinicians, like aggressiveness, depression-proneness, or impulsivity. Moreover, the major rap against the Rorschach, namely that it lacks what Lee Sechrest and Paul Meehl termed "incremental validity" in the early 1960s, has never been satisfactorily addressed. Incremental validity is the extent to which a test affords information above and beyond information that's already been collected. Sure, we can use the Rorschach to detect thinking disturbances in severely psychotic individuals, but do we really need the Rorschach to do that? If a client looks at Card 1 and sees two giraffes with boxing gloves - and gives similarly bizarre responses to other cards - there's a decent chance he or she isn't thinking straight. No matter how you look at it, Card 1 doesn't look like two giraffes with boxing gloves (well, then again...). Even those studies that have found incremental validity for Rorschach scores above and beyond other information, like self-report, suggests that this incremental validity is modest at best. So as I wrote in a letter published in today's New York Times, I'm not at all sure that making the inkblots public will harm the validity of the Rorschach very much, as I'm doubtful that many of its 100 plus scores possess all that much validity to begin with, and probably even less incremental validity above and beyond already available information.
But I'm torn because making the blots public could set a dangerous precedent. Some academic psychologists disagree with me, arguing that "science is not about secrets." They argue that science thrives on openness to critical scrutiny, and they're right. But science is also about reducing error, and making all psychological tests public will almost surely increase error by making responses easier to fake or otherwise distort, and rendering normative comparisons with other respondents well-nigh useless. The advocates of the "there should no secrets in science" view miss the crucial point that psychological tests of necessity draw on a small sample of items from a huge universe of potential items. But in using tests, psychologists attempt to draw inferences to this broader universe. So permitting respondents to see actual test items will probably compromise the ability of psychologists to draw the kinds of inferences essential to scientifically based assessment. Surely, we would not argue that the Educational and Testing Service (ETS) should make the SAT or GRE questions public before each round of test administration, or that prospective American citizens should know in advance precisely which questions they will be asked before taking their citizenship tests (similarly, although it would be a quick and easy way to boost my teacher evaluation ratings, I don't share my test questions with the students in my undergraduate psychology classes prior to their exams). Doing so confuses the sample of test items with the wider domain of test items from which it's drawn.
So to the bottom line, will the Wikipedia decision damage the field of psychological assessment? If it remains limited to the Rorschach inkblots, probably not. If it extends to a host of other psychological tests, it might.