Contrast Sets: The Importance of Anticipating Confusion

On April 14, 1994, a tragic friendly fire incident occurred in northern Iraq. Two American F-15 airplanes shot down two American helicopters, killing 26 people, primarily peacekeepers carrying out a humanitarian mission. These included 16 members of a U.N. coalition (Kurdish, Turkish, British, and French officers) as well as U.S. civilians and military officials and the aircrew. The F-15 pilots mistook the American Black Hawk helicopters for Russian-made Mi-24 “Hind” helicopters flown by the Iraqi military. No Iraqi aircraft were allowed into this no-fly zone, but the F-15 pilots didn’t expect to see any American Black Hawk helicopters, either. The F-15 pilots were just doing their job, or so they thought.

To make the confusion worse, the F-15 pilots were being observed and directed by an American AWACS (Airborne Warning And Control System) airplane, a flying command and control center. The Black Hawk helicopter pilots were also being observed and guided by an AWACS airplane. The same AWACS airplane. This accident had lots of avenues for confusion. Snook (2000) did a masterful job of unpacking the incident in his book Friendly Fire: The Accidental Shootdown of Black Hawks Over Northern Iraq.

In this post, I want to concentrate on one of these avenues for confusion—mistaking the Black Hawks for Hind helicopters. Snook explained that F-15 pilots generally fly at high altitudes and worry about threats from anti-air missiles and from enemy fighters. The F-15 pilots generally don’t fly at the lower altitudes where helicopters are found and don’t have much to fear from enemy helicopters, so the F-15 training doesn’t spend much time on identifying enemy helicopters. The training that F-15 pilots get might show a few photographs of enemy helicopters and even fewer photographs of friendly helicopters. Snook estimated that only about 5 percent of the training slides shown to F-15 pilots to prepare them for combat-depicted helicopters, and there were very few slides showing Black Hawk helicopters. I assume that the training photographs were presented one at a time.

And that’s my issue: Why show the photographs this way? Wouldn’t it make more sense to pair the photographs with their twins? Russian-made Hinds in many ways resemble American Black Hawks. So you know in advance that this is a potential confusion. And then you would show the photographs together.

But you wouldn’t give a lecture about how to tell them apart. Instead, you would let the trainees study the two and make their own discoveries about critical features. That way you would foster active learning, not the passive attempt to somehow, miraculously, commit to memory all the features of a single aircraft like an Iraqi Hind.

Snook repeats testimony from one of the F-15 pilots, “…I actually pull out the guide we have that has the silhouettes of the helicopters, and confirm that the helicopter that I’m looking at is indeed a Hind.” We can see two problems here: that the silhouettes are straight on, not the view from above. And that the pilot is only looking at the silhouette for a Hind, not also comparing it to the silhouette for a Black Hawk.

My suggestions stem from the work of Micki Chi and her colleagues—I wrote about their findings in a previous post. Chi and her colleagues advocate for encouraging trainees to self-explain. They also suggest that learning is more likely to “stick” if the learner has an active stance (e.g., letting the pilots view the contrast sets and discover the critical differences) rather than a passive stance (e.g., having an instructor recite the critical cues and expect the trainees to memorize them).

This friendly fire example is just an illustration, albeit a dramatic one, of how insufficient training can be versus how much more powerful it can be through using Chi’s suggestions—in this case, by offering contrast sets instead of just silhouettes. The example suggests how important it is to identify the decision requirements for tasks and missions instead of blindly presenting material that might be relevant, but out of context. More important, the example shows the value of trying to anticipate confusions. Borders et al. (2024) have discussed this aspect of mental models—appreciating where and how people might get confused. It certainly seems to be a part of expertise.