A Lesson in Inferential Statistics: Type I vs. Type II Errors
Sometimes the best option is the lesser of two evils.
Posted Apr 26, 2010
What do Icelandic volcanic ashes have in common with an innocent Brazilian man wrongly shot to death in a subway station by the London Metropolitan Police because they misidentified him as one of the fugitive Muslim suicide bombers? And what about God?
After six days of total ban on air travel in and out of the United Kingdom and much of northern Europe, the Civil Aviation Authority in the UK finally lifted the ban on Wednesday (21 April), resuming normal flights over the British airspace. During the ban, some European airlines, such as KLM, Air France and Lufthansa, flew their own test flights (without passengers) through the volcanic ashes and reported that it was completely safe to fly. Because the airline industry as a whole was reportedly losing $200 million a day, these airlines, after their own successful test flights, called for their respective governments to lift the ban as early as last weekend. But the ban was not lifted until three days later. After (and even during) the ban, many airline officials and stranded air travelers complained that the government’s standard for closing down the airspace was too stringent and conservative, and they demanded that the standard be relaxed. Now there are talks of some airlines and stranded passengers suing the government for financial damage. Are they right? Should the government have reopened the airspace and allowed air travel much sooner than it did?
On 22 July 2005, a Brazilian immigrant Jean Charles de Menezes was shot to death by the London Metropolitan Police officers, who mistook him as a would-be Muslim suicide bomber. This event took place one day after the failed attempts to bomb London subways by four Muslim suicide bombers, which itself happened two weeks after the successful bombings of London subways and bus on 7 July, resulting in 52 deaths. The London Metropolitan Police officers mistook de Menezes as one of the failed suicide bombers from the previous day, and shot him seven times in the head, thinking that de Menezes was about to detonate a bomb in a crowded subway car. It was quickly discovered that de Menezes was not carrying any explosives and was in no way connected to the failed bombings of the previous day. (The four perpetrators and their associates were all subsequently arrested.) Several official investigations, inquiries and inquests looked into the conduct of the police officers involved, but they were all cleared of any wrongdoing. Yet, to this day, many believe that the officers should have been held accountable for their misconduct, and some accuse the London Metropolitan Police of racism. Are they right? Should the officers involved been held criminally responsible for the tragic death of an innocent man?
Now I am going to do something that I have never done in this blog, which is to say something that everybody in the world agrees with.
It would be ideal if the government and the Civil Aviation Authority never made any mistakes in their judgment and decided to ground only those flights that were destined to crash and allowed all other flights. No one would ever complain if no safe flights were ever grounded, and only those destined to crash were grounded.
It would be ideal if the police never made any mistakes in their judgment and shot to kill only those people who were about to detonate a bomb in a crowded subway car and never killed anyone else, including completely innocent people. No one would ever complain if no innocent people were ever shot to death, and only those who were about to detonate a bomb were killed.
However, we don’t live in the ideal world. In the real world, people make decisions on the basis of insufficient information. As a result, people often make errors in judgment. Not all decisions people make will be the correct decisions. When people make errors in judgment, there are always negative consequences. The best that people can do in the imperfect real world is to minimize the negative consequences of making such errors.
There are two types of errors in judgment. There is the error of false positive of thinking that the danger is there when it isn’t. Then there is the error of false negative of thinking that the danger is not there when it is. Statisticians call the former type of errors “Type I errors” and the latter type of errors “Type II errors.” And these two types of errors often have asymmetric negative consequences.
In the case of volcanic ashes, the consequence of Type I error, which the British Civil Aviation Authority might have made, is that millions of people were stranded in their travels and airlines lost billions of dollars. The consequence of Type II error, mistakenly thinking that it is safe to fly and allowing European airlines to carry on their business as usual, is that some airplanes would crash and hundreds of people would die. There is no question which negative consequence is greater. (In all the complaints and recriminations about the ban, nobody seems to notice the miraculous fact that, in this global catastrophe of historic proportions, not a single person died. Name another natural disaster of global proportions in which no one died.)
In the case of Jean Charles de Menezes, the consequence of Type I error, which the London Metropolitan Police officers unfortunately made, is that one innocent person died. The consequence of Type II error, failing to shoot a suicide bomber who then detonates a bomb in a crowded subway, is that dozens of innocent people would die. Once again, there is no question which negative consequence is greater. People have complained about the error in judgment that the police officers actually made. But can you imagine the magnitude of complaints if the officers had made the Type II error? You can debate whether a Brazilian man should have been mistaken as one of the 21/7 Muslim suicide bombers, all of whom later turned out to be Africans. But there is no question that, as an inference system, the police procedure is the correct one.
And here is the important lessons from statistics. You cannot simultaneously decrease the probability of Type I errors and the probability of Type II errors. Any inference system that decreases the probability of Type I errors necessarily increases the probability of Type II errors. And any inference system that decreases the probability of Type II errors necessarily increases the probability of Type I errors.
Long-time readers of this blog will recognize all of this as part of the error management theory. As I discuss in the earlier posts (“Why Do We Believe in God?” Part I, Part II) which introduce the error management theory, this is precisely why humans are designed to believe in God. Humans are designed to believe in God because they are designed to be paranoid, and they are designed to be paranoid because it minimizes the risk of being killed or attacked. Humans believe in God, the Civil Aviation Authority grounds airplanes, and the police shoot an innocent person, and make mistakes in judgments in doing so, all for the same reason – the consequences of not making this mistake is even worse.