Trying to Explain the Inexplicable
We invent theories to explain noise because we discount the role of luck.
Posted Sep 23, 2016
We are hardwired to make sense of the world—to concoct explanations for anything and everything. We invent theories to explain noise because we discount the role of luck in our lives.
We see a golfer win the British Open, conclude that he is the best golfer in the world, and expect him to win the next tournament. We see a student get the highest score on a test, conclude that she is the best student in the class, and expect her to get the highest score on the next test. We see a worrisome medical test result, conclude that the patient has a disease, and prescribe a treatment.
In fact, the golfer may have been lucky; the student may have been lucky; and the patient may have been unlucky. The thing about luck—good or bad—is that it cannot be counted on to repeat, and the more extreme the luck, the less likely it is to be repeated. Then when it is not repeated, we are tempted to overreact again by inventing a seemingly plausible explanation for the inexplicable.
If the British Open champion loses the next tournament, we might conclude that he wasn’t focused. If the student with the highest test score does not do as well on her next test, we might conclude that she did not study as much. If the patient with the worrisome medical result fares better a month later, we might conclude that whatever treatment was prescribed was effective.
If, instead, we recognize that luck may have played a role, we are less likely to overreact. We will realize that the tournament winner is not necessarily the best player, that the student with the highest test score is not necessarily the best student, that the patient with a troubling medical reading does not necessarily have a disease.
We will understand that several golfers are good enough to win a tournament and several students are good enough to get the highest test score, and they take turns doing so—not because their abilities fluctuate week to week, but because their luck comes and goes. We will understand that medical test results fluctuate even if the patient’s condition does not.
The importance of luck is most easily seen in athletic competitions, like golf, where there are clear numerical measures of success and failure. There are four major golf tournaments; The Masters, U.S. Open, British Open, and PGA. There is a lot of happenstance in golf: gusts of wind, fortunate and unfortunate bounces, stray leaves, twigs, and rocks. Sometimes a ball will land on a bank and stay there, sometimes, it will roll into a lake or sand trap. Sometimes a ball will whistle through a tree, sometimes it will bounce off a branch. Sometimes a ball will bounce off a branch back onto the fairway, where the grass is cut short and the ball can be played cleanly; sometimes the ball will bounce into foot-high grass. In one 2016 golf tournament, Phil Mickelson hit an errant drive that bounced off the head of a spectator standing in the rough on one side of the fairway, and landed in the rough on the other side of the fairway. Mickelson told the spectator, “If your head was a touch softer, I’d be in the fairway.”
Yes, there is a lot of luck in golf. Here is a scatterplot of the scores for one round and the next round for the golfers who played in the Masters Tournament in 2016. There is zero correlation between the scores one round and the next.
Fifty-five golfers made the Masters cut in 2015 and 57 in 2016, but only 29 made the cut both years, which is, by itself, evidence of the importance of luck in golf. Here are the 2015 and 2016 scores for the 29 golfers who made the cut both years. (remember, low scores are better.) The correlation between the 2015 and 2016 scores is 0.38, a positive correlation, but it is loose.
For the top 15 golfers in the 2015 and 2016 Masters tournaments, there is essentially zero correlation between their scores these two years.
This is the paradox of luck and skill. When four randomly selected people play a round of golf, and their abilities vary widely, the best golfer usually wins. It is different when four of the top golfers in the world play. Their skills are nearly equal and the winner is usually determined by luck. This is why, of the 213 golfers who have won at least one of the four majors, 132 (62%) only won once. Another 37 (17%) won twice. Only 28 (13%) golfers have won more than 3 majors tournaments.
Seeking an explanation for the inexplicable, we are tempted to look for psychological explanations, to conclude that champions get lazy, or unfocused, or nervous, or spoiled, when the more convincing explanation is simply that they got lucky—and luck is fleeting.