Verified by 
Prisoner's Dilemma
The Return of Dr. Fox
Can effective speakers "seduce" an audience into believing they are experts?
Posted May 5, 2014
Did you ever hear of the "Dr. Fox Effect?"
In a classic experiment conducted by Donald Naftulin, John Ware, and Frank Donnelly in the early 1970s, three separate groups of professionals attended a lecture titled, "Mathematical game theory and its application to physician education.” The professionals, including psychiatrists, psychologists, social workers, and educational administrators, were all attending continuing education training conferences.
In the description about the speaker, they were told that "Dr. Myron L. Fox" was a student of John von Neumann and an authority on game theory in his own right. As the author of books and articles on game theory and mathematics, Dr. Fox's expertise was, to put it mildly, considerable.
The reality was very different, however. The speaker introducted to the as "Dr. Fox" was actually a paid actor whose only knowledge of game theory consisted of one Scientific American article he had read to prepare for the role. Naftulin, Ware, and Donnelly had coached him to present his lecture "with an excessive use of double-talk, neologisms, non sequiturs, and contradicting statements.” In fact, many of the people in the audience likely knew more about game theory than he did.
Despite the largely meaningless content, the lecture was presented in a lively and humourous speaking style with the speaker interacting warmly with the audience. After giving the lecture, audience participants filled out a teacher evaluation questionnaire providing yes/no responses about the speaker' performance such as "Did the speaker show his interest in the topic?", "Did he present his material in an interesting way?", and "Was his material well-organized?".
After evaluating their results for the fifty-five respondents who provided questionnaires, Naftulin and his colleagues concluded that the overwhelming majority of people submitting responses described the speaker favourably. In the 1973 paper that they wrote about the experiment titled: "“The Doctor Fox Lecture: A Paradigm of Educational Seduction”, Naftulin et. al. concluded that the actor's expressive lecturing style basically "seduced" the people hearing the lecture into believing that they had actually learned something substantial.
Since the paper's publication, the experiment has been repeatedly replicated and the "Dr. Fox effect" is still widely cited despite heavy criticism by many researchers. The criticism often focuses on the lack of a proper control group, the use of a "yes/no" scale that might have led to a "yes-saying" response bias, and the lack of any measurement of actual learning.
Despite these criticisms, the original experiment is often invoked as evidence against the use of student evaluations to rate teacher effectiveness. Since student evaluation of teachers (SET) is still widely used to rate teacher effectiveness and can play a role in hiring decisions, salary negotiations, and budget decisions, understanding how and why the "Dr. Fox effect" occurs remains important today.
In a new research study published in the Journal of Educational Psychology, Eyal Peer of Carnegie Mellon University and Elisha Babad of Hebrew University of Jerusalem took the unusual step of replicating the original 1973 experiment with a new generation of students. Using 247 undergraduates from different behavioural science courses (78.9 percent female), Peer and Babad attempted to follow the original design as closely as possible. After seeing the original video of "Dr. Fox" lecturing, the students then completed questionnaires rating the speaker.
While one group completed a questionnaire similar to the one used by Naftulin, Ware, and Donnelly, other groups completed slightly different questionnaires to test whether the "yes/no" format may have influenced the results. Also, instead of positively worded items ("Did the speaker seem interested in his subject?"), some questionnaires contained negatively worded items ("Did the speaker seem uninterested in his subject?").
Another experimental condition was the use of cognitive remedies to ensure the participants responded as honestly as possible. This involved providing this "objectivity instruction" warning: "Please pay attention. Past research demonstrated a type of bias in which questionnaire respondents tend to agree with any statement presented to them. Please try as best as you can to avoid this bias in filling out the questionnaire, and respond according to your true feelings."
Results showed that the majority of students seeing the video described the speaker favourably. Even in the other conditions, wording of the questionnaire and using specific warnings about possible bias made little difference in the overall results. Based on these results, the "Dr. Fox effect" appears fairly strong. But what would happen if the speaker were not presented as an eminent authority? Would there still be a "Dr. Fox effect"?
A second study, almost identical to the first, was carried out but the experimenters removed the first thirty seconds describing Dr. Fox's credentials. In all experimental conditions, the Doctor Fox effect still remained strong whether or not the audience knew anything about the speaker's background.
As for the third study, which included students who were actually familiar with game theory and who would presumably not be fooled by the speaker's pretend lecture, graduate students who knew more about game theory than the speaker were still just as likely to rate him in a positive way as freshman university students.
But does any actual learning take place regardless of whether or not people in the audience rate the speaker in a positive way? To test this, Peer and Babad added an additional question to the questionnaire asking whether the people who attended the lecture felt that they had actually learned anything. What they found was that even students who reported that Dr. Fox was a highly effective speaker did not actually think that they had learned anything from the lecture.
Whether the students knew anything about game theory or not, there seemed to be no real relationship between self-reported learning and rating of speaker effectiveness. Even when respondents were asked whether the speaker "stimulated their thought", only slightly more than half agreed with that statement.
What this result apparently demonstrates was that there was no actual "seduction" as Naftulin, Ware, and Donnelly reported in their original study. Even when people hearing the Dr. Fox lecture rated him highly as an effective speaker, they were apparently well aware that they weren't learning anything constructive. While Naftulin and his fellow researchers argued that people hearing the bogus lecture actually believed they were learning something constructive, this new study shows the exact opposite. People listening to an engaging and entertaining speaker may enjoy what they are hearing, but they appear well aware that the lecture isn't teaching them anything important.
Though the Dr. Fox experiment is frequently cited as proof of educational "seduction", the original study by Naftulin, et al. is still as controversial as ever. That there really is a "Dr. Fox effect" has been repeatedly demonstrated over the years though the implications are likely not as dire as educational researchers suggest.
Warm and charismatic speakers may entertain an audience but being a good teacher needs far more than that. However many "Dr. Fox"-type lecturers there are out there, students seem well aware that being entertaining is not as important as being competent to teach a subject so that students can actually learn from them.
Ultimately, there is no such thing as "educational seduction" and many of the researchers who have claimed otherwise over the years likely did so without really questioning the original 1973 study. In a fitting closing statement, Peer and Babad quote Abraham Lincoln (and Bob Marley) by pointing out: "“You can fool some of the people some of the times, but you cannot fool all the people all the time!”
