The recipe for this type of research is this: Identify the top companies of the time using performance data such as profitability or market share. Then, collect data about business and management practices by interviewing executives and mining publications and other media sources for information. Then, search for patterns and sort the data into categories. Then, point to the association between the practices data and the performance data. Finally, suggest that the former cause the latter.

If the sample only includes top-performing companies, the association cannot be expressed as a correlation, for a correlation between two variables requires variation in both. A better approach is to sample companies more broadly. At least each high-performing company can be paired with an average-performing one that is similar in other relevant respects, such as size and product line. This yoking of companies guards against the error of comparing apples and oranges. This was the approach taken in Built to Last (Collins & Porras, 1994) and other books that came in its wake.

Excellence and Built yielded similar results and recommendations. They found that the top companies had, among other things, [1] a strong corporate culture, [2] a bias for doing rather than mulling, [3] a focused strategy, and [4] persistence in their goal-pursuit. They hence recommended that thou shalt go out and do likewise, and all will be well.

All was not well. Time and again, when companies were re-evaluated a few years later, most top ones were no longer top and many below-average companies had risen. After ten years, the formerly excellent companies were slightly below average on average.

Rosenzweig (2007) describes these findings in The Halo Effect. His book is an effective antidote to the typical business research and writing, and I want to tell you more about it.

Halo raises three questions: [1] What are the reasons for the unpredictability of sustained corporate success? [2] Why do business researchers stubbornly continue to crank out books like Excellence or Built? [3] If the conventionally offered results and recipes are unreliable, what are we to do instead? Let's consider each question in turn.

The elusiveness of sustained corporate success

Rosenzweig put his answer in his book's title: The Halo Effect with a capital H. Harkening back to Thorndike (1920), Rosenzweig shows that in most of the major studies since Excellence, corporate performance ratings were not independent of one another. There was, in Thorndike's terms, "a constant error in psychological ratings." Corporations rated high in innovativeness were also rated high in action orientation, which were also rated high in strength of corporate culture etc. The Halo effect means that these features are more highly correlated with one another at the level of human judgment than they would be if they were measured objectively.

But the real problem with the Halo effect is not the exaggerated consistency of its elements, but that the ratings of all the constituent variables are affected by the known outcome, i.e., the company's success. There is a huge outcome bias. People naturally infer that the victors in a competitive world must have done something right, and, lacking insight into the specifics of the target's (i.e., the organization and its leadership) behavior, they rate it highly on all desirable attributes.

Outcome bias is a threat to the validity of all retrospective judgments. Suppose you have a choice between a certain gain of $200 and a gamble yielding a gain of $400 with a probability of .8 and a loss of $100 with a probability of 0.2. The gamble has a higher expected value ($300) than the sure gain, and so you might take it if maximizing expected value is what your are interested in. Yet, 20% of you will lose and probably feel regret. This regret is irrational; it stems from the outcome bias. For the backward looking human, it is hard to acknowledge that the risk-accepting choice was the right one at the time. Rational decisions do not guarantee success; they only make it more likely in the aggregate.

Outcome bias would not be a problem if corporate success were indeed stable. If past, current, and future performance were highly correlated, basing one's judgment on past performance would be the smart thing to do. But it isn't. Rosenzweig points to many factors that disrupt the past-future correlation. One such factor is the competitiveness of the economy. Even a company that does everything right by the book (Excellence or Built) can be surpassed by companies that do even righter - companies that apply the recipe even more conscientiously. Competition, and other correlation-disrupting factors, such as rapid or unprognosticated technological change (not to mention Schumpeter's creative destruction), degrade our ability to predict who will be tops tomorrow. Once we know that the correlation between success rankings over time is far less than perfect, we know (rather: we should know) that we are in for a regression effect. Over time, the performance of companies will gravitate toward the average, and this trend will be stronger the more extreme the company's score is at the time we are looking. This means that research presented in works like Excellence or Built maximizes the regression effect precisely because it is looking at the toppest companies at the time of study. Aside from rare exceptions (which must also happen with a derivable probability) these companies have no place to go but down. As Rosenzweig puts it, "Nothing recedes like success" (italics his).

Why do case-study-type books keep coming?

Researching the best is a seductive strategy. It is intuitive. If you want to understand genius, you need to see what the geniuses do. It is also intuitive to think that understanding the past is the same as being able to predict the future (but it ain't). Rosenzweig shows that over generations of books of the Excellence type, research methods improved at the margins, but failed to break the back of the fundamental error. Instead of sampling companies based on present performance and looking back to find what makes them unique, a scientific strategy requires either random sampling or sampling representatively with regard to variables of interest, and then to see which of these variables predict success in the future. With this method, there is a shot at causal understanding with prediction.