You have likely heard that experts are notoriously bad at forecasting. This insight comes from the work of Philip Tetlock who has written a brand new book titled Superforecasting: The Art and Science of Prediction.

There are approximately 20,000 intelligence analysts working in the United States today for various agencies, “assessing everything from minute puzzles to major events such as the likelihood of an Israeli sneak attack on Iranian nuclear facilities or the departure of Greece from the Eurozone.” When intelligence agencies get things wrong (e.g., weapons of mass destruction in Iraq) or fail to predict a tragic event (e.g., 9/11), they get beat up in the media and by politicians. Forecasting these events is extremely complex and can never be done with certainty. Rather than slinking away and covering their asses, beginning in 2011, the Intelligence Advanced Research Projects Activity (IARPA), the DARPA of the intelligence community, created a tournament in an attempt to learn how their forecasts could be improved. In the beginning, five teams competing by generating forecasts in response to the types of tough questions that intelligence analysts routinely face. One of those five teams was the Good Judgment Project (GJP), organized by Tetlock and his research (and life) partner Barbara Mellers. After two years, the GJP team was doing so much better than the others, IARPA dropped the others and focused entirely on what they could learn from the GJP team. Over four years these forecasters responded to nearly 500 questions about world affairs, with predictive timelines extending from just over a month to a little less than a year. During the course of this tournament, Tetlock and Mellers were able to hone the team into a select group that they called “superforecasters.” The book Superforecasters presents their findings, and they’re extraordinary.

To determine the best forecasts, predictions had to be made in clear and unambiguous terms that could be later compared to actual events. Also, the effectiveness of forecasters had to be measured through hundreds of forecasts to confidently identify the best. Once identified, these superforecasters were carefully observed to find out what separated them from the others. Here, in a nutshell, is a summary of Tetlock’s and Mellers’ findings:

Superforecasting does require minimum levels of intelligence, numeracy, and knowledge of the world, but anyone who reads serious books about psychological research [such as Tetlock’s] probably has those prerequisites. So what is it that elevates forecasting to superforecasting? …What matters most is how the forecaster thinks… Broadly speaking, superforecasting demands thinking that is open-minded, careful, curious, and—above all—self-critical. It also demands focus. The kind of thinking that produces superior judgment does not come effortlessly. Only the determined can deliver it reasonably consistently, which is why our analyses have consistently found commitment to self-improvement to be the strongest predictor of performance. (p. 20)

Do not be satisfied with this summary. Many of the qualities that produce effective forecasts can be developed. If you’re reading this blog, chances are they’re within your reach. Read this fascinating book for yourself to discover and understand what Tetlock and Mellers found.

Take care,