CHAPTER 9
How to Harness Experts
Many managers are inclined to rely on experts. Before we evaluate that practice, we need to say a few words about expertise. It’s obvious that the term can be understood in many ways. Here we mean expertise in the sense of being able to make winning bets on the future. This ability could be shown implicitly through the capacity to come up with a good design—as, for example, by building a bridge or another structure that performs well in an uncertain future of storms and erosion, or by designing an investment strategy that really is risk-proof. More relevant to our concerns here is the ability to make accurate forecasts about uncertain future events.
Unfortunately, very few so-called experts can make good predictions. When most experts are put to a careful test, they prove to be much less competent than popular opinion expects (or than they themselves claim). In most professions, telling a pleasing and convincing story, rather than making accurate predictions, is the primary evidence relied on to assign someone the label expert. Often those stories are told after the fact, not in advance, and when they are told in advance, they tend to have more superficial plausibility than factual accuracy.
Political pundits who have been “scored” on the accuracy of their election forecasts are generally found to perform at chance or even worse.1 Similar conclusions apply to popular stock-pickers, many of whom turn out to be a bit like astrologers. A pathbreaking book by the social scientist Philip Tetlock finds that for a wide range of events, professional experts turn out to be pretty terrible forecasters.2 And in fact, the book provides data showing that the popularity and public visibility of an expert is negatively correlated with his or her accuracy. Sadly, it seems for pundits, confidence trumps accuracy.
To be sure, there are some clear cases of documented expertise, including weather forecasters, physicists (forecasting sunspots and other astronomical phenomena), chess and bridge experts, and some professionals in branches of medicine and financial auditing.3 The key to knowing when experts can make accurate predictions is to understand two factors: the conditions surrounding their training and the environment in which they make their forecasts.
One form of expertise derives from a sound scientific understanding of the particular domain in which judgments are made. In physics, meteorology, engineering, and some areas of medicine and finance, there are empirically validated theoretical frameworks and well-established forecasting and diagnostic methods. The same is true in sports, especially baseball, where experts, armed with statistical methods, can make pretty good predictions about player performance (as celebrated and made famous by Moneyball). When experts are trained in the relevant theories and methods, perhaps assisted by analytic software programs, it is reassuring that they can turn the vast foundation of scientific wisdom into accurate predictions.
Nate Silver’s distinctive ability to predict political election outcomes is an example of this form of expertise.4 Rather than responding on the basis of his intuition, experiences, or prejudices about who should win an election, Silver relies on hundreds of years of statistical theory and practice, interpreting the results of polls to make forecasts. Silver’s very public success has led to a small (and entirely justified) burst of indignation about the lousy track records of many other publicly identified political experts.5
A second form of expertise is experiential, not theory based, and is derived from training, often via apprenticeships in which the learner gets quick and precise feedback while making many practice judgments. This kind of expertise takes the form of “knowing what to look for” and is often summarized in advice about how to make assessments.6
For example, most livestock judges are extremely accurate when asked to estimate the market value or breeding quality of a prize ox or sow. They learn this skill through an extensive apprenticeship in which they make thousands of such judgments, are required to defend each judgment, and then receive immediate feedback from more-senior experts. A similar training regimen applies to certified weather forecasters, who are also assisted by expert software and long study of meteorological theory. In these cases, careful behavioral analysis has found that the judge’s expertise is best described as knowing which particular aspects of the situation to study and then how to combine the information from those cues into a judgment.
The third form of expertise is also based on experience with many thousands of example cases, again followed by quick, precise feedback. This form of expertise was first identified as underlying the remarkable skills of chess players by the Nobel Prize–winning behavioral scientist Herbert Simon and his colleague Bill Chase.7 These scientists were intrigued by the question of chess expertise (what separates the world-class players from the novices) and especially by the puzzle of why grandmasters’ skill hardly declined in speed-chess matches, where moves are made in seconds, not minutes.
Their conclusion was that the thousands of hours studying past games led to the development of an elaborate expert memory store of tens of thousands of mental snapshots of chessboard configurations with stored best-move “answers” for each snapshot. Many other examples of this form of expert capacity have been identified in subsequent research in fields ranging from sports skills to medical wisdom.8 In addition to the athletic ability to execute offensive strategies, a great football quarterback has a mental library of visual images that instantly identify configurations of defenders to support that uncanny ability to read the defense and select targets for pass reception. Champion tennis players have the same kind of mental library. A similar snapshot-matching process underlies the diagnostic skills of experienced dermatologists and radiologists.9
These three thinking skills—theory-based reasoning, knowledge about which valid cues to use, and case-based memory-matching—are the primary forms of expertise identified by behavioral scientists. We know of no other credible forms of verified judgmental expertise.
There are two important points to remember. First, a reliable capacity to predict the future with better-than-modest accuracy is very rare. Second, some highly specific forms of expertise do exist. So, if you are attempting to leverage intelligence out of a collective, be sure you start with a sample of verified experts.
If you start with such experts, you might well end up doing a lot better than if you take a general sample of the population and rely on its average answer. Real experts will almost certainly do better than statistical averages of amateurs. Even here, however, there are some important lessons for wise groups to keep in mind.
Enlisting Expertise
We should now understand that performance depends on the competence of the individuals making the relevant judgments. Suppose that we could find real experts in estimating the weight of oxen or in counting jelly beans; suppose too that we understand expertise to be the ability to make accurate assessments. If so, then these (admittedly weird and obsessive) experts would, by definition, do better than statistical averages. In the real world, we must often choose between a small group of people, each with a large amount of information, and a large group of people, each with a small amount of information. Everything depends on the levels of expertise; sometimes the large group is worse.
One thing is clear: whether we are dealing with experts or non-experts, it makes sense to obtain a statistical answer from a group of them, rather than to depend on one or a few. If experts are likely to be right, a statistical group of experts should have exactly the same advantage over individual experts as a statistical group of ordinary people has over ordinary individuals. Many expert minds are likely to be better than a few.
A great deal of evidence supports this claim.10 In a series of thirty comparisons, statistical groups of experts had 12.5 percent fewer errors than individual experts on forecasting tasks involving such diverse issues as cattle and chicken prices, real and nominal gross national product, survival of patients, and housing starts.11 So too, statistical groups of experts significantly outperformed individual experts in predicting the annual earnings of firms, changes in the American economy, and annual peak rainfall runoff.12 For groups, the implication is straightforward. In forecasting expert Scott Armstrong’s words, “organizations often call on the best expert they can find to make important forecasts. They should avoid this practice, and instead combine forecasts from a number of experts.”13 Leaders of companies and nations, please take note.
For political polling, it has become standard practice to combine several poll results and, just as with individual human forecasters, to rely on the average or median, rather than to select one or two. This practice has had excellent results. There is a general lesson here for predictions about electoral outcomes. When forecasters lack enough different polls on the same question, they should instead use multiple indicators—a range of diverse questions that tap into the same underlying factor. The answer might be averaged into some kind of composite, one that is likely to be more reliable than individual items.
No Chasing
These points bear on the decisions of businesses and governments alike. The problem is that people, including leaders, are usually unwilling to use averages of experts. Instead, the norm is to chase the expert, in an effort, usually futile, to beat the average by identifying the single most accurate thinker. This unfortunate habit might well be a by-product of the ubiquitous hindsight bias. After the fact, we know exactly who was closest to the truth, and this knowledge makes us forget that we had no clue about who was the most accurate before the fact.
Chasing the expert is usually a terrible idea, as a study by Albert Mannes and his colleagues demonstrated.14 They asked Duke University MBAs to play the financial forecasting game staged twice a year by the Wall Street Journal. The public contest pitted economic experts against one another in forecasting salient economic trends such as interest rates and unemployment statistics. The MBAs were asked to make forecasts for twenty-four periods. They were able to rely on either the best single expert (according to past predictions) or an average of the predictions of all the experts.
The results were dramatic: 80 percent of the MBAs picked the single best expert. How did chasing the expert serve them? Not well. Under all experimental conditions, chasing the expert performed significantly worse than relying on the average of all experts.
Still, if we decide to consult all the experts, is there a better way to combine their judgments than a simple average? Here we get to the frontiers of behavioral science wisdom. A helpful source is a large-scale research project funded by the Office of the Director of National Intelligence.15 Several teams of researchers addressed the question of how to improve intelligence analysis in forecasts of political, diplomatic, and economic events. The emerging conclusion is that there is some improvement if we weight individual experts by their track record of prior forecasts for similar events. So the Duke MBAs were right to think the best expert was the most informative, but they overreacted and ignored the information contained in other experts’ estimates.
A collateral finding from the intelligence-forecasting tournament was that the experts’ self-evaluations of their own judgments were not useful. Weighting the experts’ judgments by each expert’s own self-confidence did not improve accuracy of the composite; experts lacked subjective insights into their own performance. Only weighting by objective “batting averages” of measured past accuracy was helpful.
The lesson is clear: do not be misled by expert bravado or by an expert’s own sense of how he or she is doing. Evidence is a much better guide than an impressive self-presentation.