Mastering the Game of Imperfect Forecasting

Mastering the Game of Imperfect Forecasting

Before the season started, English Premier League football club Leicester City were 5,000-to-one longshots to win the title. In fact, they were the odds-on favorite to finish at the bottom of the table and therefore dropped to a lower division. The team was largely made up of low cost acquisitions of has beens, never weres and maybe will bes. Their primary aspiration had been merely to avoid relegation.

Leicester’s ownership was ridiculed for hiring the 64-year-old Claudio Ranieri as manager last July. Despite having managed some of Europe’s top clubs, he had never won a championship and he had been out of work since the previous year when he was fired by the Greek national team after an improbable loss to the Faroe Islands. His only job in the Premier League had ended 11 years earlier.

But deft management, team spirit, a tremendous work ethic, surprising player development, nearly every transaction working out, nearly every player having his best year ever and a remarkable lack of injury combined to create what is likely the greatest longshot victory in the history of sport. The alleged “experts” — who picked the usual suspects to win it all — failed.

History has provided a long list of similar forecasting failures. Analyst Clifford Stoll argued that “no online database will replace your daily newspaper.” Bob Metcalfe, an electrical engineer widely credited with the invention of Ethernet technology, predicted that the internet would “in 1996 catastrophically collapse.” Federal Communications Commission commissioner T.A.M. Craven stated in 1961 that “There is practically no chance communications space satellites will be used to provide better telephone, telegraph, television or radio service inside the United States.”

Marconi predicted that the “wireless era” would make war ridiculous and impossible. Decca Records rejected the Beatles because they didn’t like the group’s sound and thought guitar music was on the way out. Every other studio in Hollywood but one turned down “Raiders of the Lost Ark” before Paramount agreed to make it and it became one of the highest-grossing films of all time.

Thomas Bell, president of the Linnean Society of London, summing up the year 1858 (which included the announcement of Charles Darwin’s theory of evolution by natural selection), stated: “The year which has passed has not, indeed, been marked by any of those striking discoveries which at once revolutionize, so to speak, the department of science on which they bear.”

In April 1900, the great physicist Lord Kelvin proclaimed that our understanding of the cosmos was complete except for two “clouds” — minor details still to be worked out. Those clouds had to do with radiation emissions and with the speed of light, and they pointed the way to two major revolutions in physics still to come: quantum mechanics and the theory of relativity.

At the World Economic Forum in 2004, Bill Gates predicted that, “Two years from now, [email] spam will be solved.” And nearly every political pundit gave Donald Trump no chance to win the Republican presidential nomination.

Most of the alleged experts making market predictions are highly educated, vastly experienced, and examine the vagaries of the markets pretty much all day, every day, Yet they too are wrong a lot — pretty much all the time in fact. Why are we so bad at forecasting?

Causal Complexity

As Daniel Kahneman and Amos Tversky so powerfully pointed out, we evolved to make quick and intuitive decisions for the here and now ahead of careful and considered decisions for the longer term. We intuitively emphasize (per anthropologist John Tooby) “the element in the nexus that we [can] manipulate to bring about a favored outcome.” Thus, “the reality of causal nexus is cognitively ignored in favor of the cartoon of single causes.” In short, whenever we try to figure out complex future outcomes we enter dangerous territory with disaster lurking everywhere.

Even when we recognize the fallacy of thinking in terms of single, linear causes (Fed policy, market valuations, etc.), the markets are still too complex and too adaptive to be readily predicted. There are simply too many variables to predict market behavior with any degree of detail, consistency or competence. Unless you’re Seth Klarman or somebody like him (none of whom is accepting capital from the likes of us), your crystal ball almost certainly does not work any better than anyone else’s.

All that said, the idea that we can live our investing lives forecast-free is as erroneous as the market predictions that are so easy to mock. As Cullen Roche repeatedly emphasizes, “any decision about the future involves an implicit forecast about future outcomes.” As Philip Tetlock wrote in his wonderful book, “Superforecasting: The Art and Science of Prediction”: “We are all forecasters. When we think about changing jobs, getting married, buying a home, making an investment, launching a product, or retiring, we decide based on how we expect the future to unfold.”

Improving the Odds

It’s a grand conundrum for the world of finance — we desperately need to make forecasts in order to serve our clients but we are remarkably poor at doing so.

The key then, as Roche has argued, is that we should shun low probability forecasts. By contrast, “Superforecasting” points to (and laughs at) the general inaccuracy of financial pundits at CNBC, whose forecasts are low probability ones of the highest order. As Jon Stewart famously remarked, “If I’d only followed CNBC’s advice, I’d have a million dollars today — provided I’d started with a hundred million dollars.”

The central lessons of “Superforecasting” can be distilled into a handful of directives. Base predictions on data and logic, and try to eliminate personal bias. Working in teams helps. Keep track of records so that you know how accurate (or inaccurate) you (and others) are. Think in terms of probabilities and recognize that everything is uncertain. Unpack a question into its component parts, distinguishing between what is known and unknown, and scrutinizing your assumptions. Recognize that the further out the prediction is designed to go and the more specific it is, the less accurate it can be.

In other words, we need rigorous empiricism, probabilistic thinking, a recognition that absolute answers are extremely rare, regular reassessment, accountability and an avoidance of too much precision. Or, more fundamentally, we need more humility and more diversity among those contributing to decisions. We need to be concerned more with process and improving our processes than in outcomes, important though they are. “What you think is much less important than how you think,” says Tetlock. Superforecasters regard their views “as hypotheses to be tested, not treasures to be guarded.” As Tetlock told Jason Zweig of The Wall Street Journal, most people “are too quick to make up their minds and too slow to change them.”

Most importantly, perhaps, Tetlock encourages us to hunt and to keep hunting for evidence and reasons that might contradict our views and to change our minds as often and as readily as the evidence suggests. One “superforecaster” went so far as to write a software program that sorted his sources of news and opinion by ideology, topic and geographic origin, then told him what to read next in order to get the most diverse points of view.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s