I have just come back from the Association of Psychological Science (APS) meetings, where I heard an interesting symposium chaired by Philip Ackerman (Georgia Tech). In that symposium Angela Duckworth (U. Pennsylvania) presented a paper with a message that I think ought to be widely distributed, for it touches on what some think is a paradox in the field.
(Truth in packaging..Dr. Duckworth's thesis impressed me because I agree with it! See my book HUMAN INTELLIGENCE, and the part about testing. I wish I had heard Dr. Duckworth before the book was published, for she said the ideas better than I.)
The thesis..which applies to educational achievement testing and personality testing as well as intelligence testing..is that we can only do so much with what Robert Mislevy (U. Maryland) has called the "drop from the sky" method of testing. In "drop from the sky" testing an examiner poses a set of questions to the examinee, and does so out of context of the examinee's normal life, and does so in a limited time. (Timed vs. Power tests are not the issue here. The typical power test still has to be completed in a testing session of from one to three hours.) Binet, and over a century of research after him, has shown that you can evaluate some of a person's cognitive capabilities by the "Drop from the sky" method. You can evaluate a person's superficial knowledge of, say, the American Civil War by asking questions like "who the leaders were" and "who won the battle at Gettysburg?" What you can't do is ask a question like "What were the mix of social, economic, and religous issues that led to the war?" You can also evaluate a person's capacity for attention and memory...within brief limits. You cannot evaluate a person's ability to order his or her efforts to achieve different goals (including studying over partying), or the person's ability to look at things from various perspectives. More generally, you cannot test any capability that is only displayed over time. Yet these capabilities are extremely important to a person's cognitive capabilities. Here's a historic example.