Why is Jeremy Lin so good? Maybe he's not. He might just be lucky. But his success illustrates our psychological inability to come to grips with an iron law of statistics: regression to the mean.
But first, there's no question Lin is hot right now. Linsanity is the biggest story in American sports. Last week he led the Knicks to a win over the Lakers with 38 points; last night he hit the game winning three-point shot with .5 seconds left in th game to beat Toronto.
Lin is a Harvard-educated Asian-American, which is unusual for the NBA. But that doesn't mean he can't be good; the surprising part is that he came from nowhere. He was undrafted out of college and then cut by the lowly Golden State Warriors before warming the Knicks bench. But no longer.
Even more surprising is the way the stock market is reacting: The Knicks' stock price is at a record high, and Lin seems to be the cause. That's not Linsanity. It's insanity.
Unfortunately for Lin and the Knicks' investors, a statistical monster is lurking around the corner: regression to the mean.
Regression to the mean is based on two assumptions. First, Lin has a certain level of basketball ability. Second, people can play above (or below) their level for a while, but eventually they'll settle back in to their average. The idea can be summarized as follows:
In other words: Your next game will be average. If Lin's basketball ability is 541 (on average; this is a made up number), he'll probably be a 541 player in his next game.
It's just that lately Lin's been playing like a 900 player. Based on regression to the mean, he'll soon be playing like a 541 player again. In other words, he's bound to get "worse" sooner or later. (Or maybe he really is a 900 player, in which case I'm totally wrong.)
Regression to the mean is why players who are named Rookie of the Year often have a "sophomore slump." They don't get worse in their second year. It's just that the best rookie in the league tends to be a good player who's having a great year. If he follows it up as a good player who has an avearge year, which regression to the mean says he probably will, then he'll be perceived as an underachiever who is in a slump.
That doesn't make sense
Regression to the mean is an unnatural concept to the human mind. It is in our nature to look for cause and effect relationships: We instantly assume that one thing leads to another. So when Lin is playing well, we explain it based on actual causes, like his newfound confidence, his fit with the Knicks system, etc. Looking online, you'll find a million theories. If Lin starts playing worse, more causes will be hypothesized: Did the pressure get to him? Did the opponents start focusing their defenses on shutting him down?
These explanations could be right. But what if there is no cause? Lots of evidence says his hot streak could just be a random statistical aberration (see the hot-hand phenomenon). In other words, he's been lucky lately. And there need not be a cause when he cools off either; it's just random regression to the mean. These explanations are not intuitive (and no one in the media talks about them), but they may be right.
When he does cool off, it's not because he's getting worse. He's just stopped playing better than he is. (If it seems obvious that he can't keep this up, it's not. Just check the stock market. Big money is betting on him keeping up the show.)
Sports and investment seem like two patches of our cultural landscape where people should understand statistics. Apparently it's not that simple. Regression to the mean is hunting Jeremy Lin. I hope I'm wrong, but I doubt it.