Shaun Botterill/Getty Images

Before the season started, English Premier League football club Leicester City were 5,000-to-one longshots to win the title. In fact, they were the odds-on favorite to finish at the bottom of the table and therefore dropped to a lower division. The team was largely made up of low cost acquisitions of has beens, never weres and maybe will bes. Their primary aspiration had been merely to avoid relegation.

Leicester’s ownership was ridiculed for hiring the 64-year-old Claudio Ranieri as manager last July. Despite having managed some of Europe’s top clubs, he had never won a championship and he had been out of work since the previous year when he was fired by the Greek national team after an improbable loss to the Faroe Islands. His only job in the Premier League had ended 11 years earlier.

But deft management, team spirit, a tremendous work ethic, surprising player development, nearly every transaction working out, nearly every player having his best year ever and a remarkable lack of injury combined to create what is likely the greatest longshot victory in the history of sport. The alleged “experts” — who picked the usual suspects to win it all — failed.

History has provided a long list of similar forecasting failures. Analyst Clifford Stoll argued that “no online database will replace your daily newspaper.” Bob Metcalfe, an electrical engineer widely credited with the invention of Ethernet technology, predicted that the internet would “in 1996 catastrophically collapse.” Federal Communications Commission commissioner T.A.M. Craven stated in 1961 that “There is practically no chance communications space satellites will be used to provide better telephone, telegraph, television or radio service inside the United States.”

Marconi predicted that the “wireless era” would make war ridiculous and impossible. Decca Records rejected the Beatles because they didn’t like the group’s sound and thought guitar music was on the way out. Every other studio in Hollywood but one turned down “Raiders of the Lost Ark” before Paramount agreed to make it and it became one of the highest-grossing films of all time.

Thomas Bell, president of the Linnean Society of London, summing up the year 1858 (which included the announcement of Charles Darwin’s theory of evolution by natural selection), stated: “The year which has passed has not, indeed, been marked by any of those striking discoveries which at once revolutionize, so to speak, the department of science on which they bear.”

In April 1900, the great physicist Lord Kelvin proclaimed that our understanding of the cosmos was complete except for two “clouds” — minor details still to be worked out. Those clouds had to do with radiation emissions and with the speed of light, and they pointed the way to two major revolutions in physics still to come: quantum mechanics and the theory of relativity.

At the World Economic Forum in 2004, Bill Gates predicted that, “Two years from now, [email] spam will be solved.” And nearly every political pundit gave Donald Trump no chance to win the Republican presidential nomination.

Most of the alleged experts making market predictions are highly educated, vastly experienced, and examine the vagaries of the markets pretty much all day, every day, Yet they too are wrong a lot — pretty much all the time in fact. Why are we so bad at forecasting?

The great Russian novelist Leo Tolstoy gets to the heart of the matter when he asks, in the opening paragraphs of Book Nine of “War and Peace”: “When an apple has ripened and falls, why does it fall? Because of its attraction to the earth, because its stalk withers, because it is dried by the sun, because it grows heavier, because the wind shakes it, or because the boy standing below wants to eat it?” With almost no additional effort, today’s scientists could expand this list extensively.

Causal Complexity

As Daniel Kahneman and Amos Tversky so powerfully pointed out, we evolved to make quick and intuitive decisions for the here and now ahead of careful and considered decisions for the longer term. We intuitively emphasize (per anthropologist John Tooby) “the element in the nexus that we [can] manipulate to bring about a favored outcome.” Thus, “the reality of causal nexus is cognitively ignored in favor of the cartoon of single causes.” In short, whenever we try to figure out complex future outcomes we enter dangerous territory with disaster lurking everywhere.

Even when we recognize the fallacy of thinking in terms of single, linear causes (Fed policy, market valuations, etc.), the markets are still too complex and too adaptive to be readily predicted. There are simply too many variables to predict market behavior with any degree of detail, consistency or competence. Unless you’re Seth Klarman or somebody like him (none of whom is accepting capital from the likes of us), your crystal ball almost certainly does not work any better than anyone else’s.

All that said, the idea that we can live our investing lives forecast-free is as erroneous as the market predictions that are so easy to mock. As Cullen Roche repeatedly emphasizes, “any decision about the future involves an implicit forecast about future outcomes.” As Philip Tetlock wrote in his wonderful book, “Superforecasting: The Art and Science of Prediction”: “We are all forecasters. When we think about changing jobs, getting married, buying a home, making an investment, launching a product, or retiring, we decide based on how we expect the future to unfold.”

Improving the Odds

It’s a grand conundrum for the world of finance — we desperately need to make forecasts in order to serve our clients but we are remarkably poor at doing so.

The key then, as Roche has argued, is that we should shun low probability forecasts. By contrast, “Superforecasting” points to (and laughs at) the general inaccuracy of financial pundits at CNBC, whose forecasts are low probability ones of the highest order. As Jon Stewart famously remarked, “If I’d only followed CNBC’s advice, I’d have a million dollars today — provided I’d started with a hundred million dollars.”

The central lessons of “Superforecasting” can be distilled into a handful of directives. Base predictions on data and logic, and try to eliminate personal bias. Working in teams helps. Keep track of records so that you know how accurate (or inaccurate) you (and others) are. Think in terms of probabilities and recognize that everything is uncertain. Unpack a question into its component parts, distinguishing between what is known and unknown, and scrutinizing your assumptions. Recognize that the further out the prediction is designed to go and the more specific it is, the less accurate it can be.

In other words, we need rigorous empiricism, probabilistic thinking, a recognition that absolute answers are extremely rare, regular reassessment, accountability and an avoidance of too much precision. Or, more fundamentally, we need more humility and more diversity among those contributing to decisions. We need to be concerned more with process and improving our processes than in outcomes, important though they are. “What you think is much less important than how you think,” says Tetlock. Superforecasters regard their views “as hypotheses to be tested, not treasures to be guarded.” As Tetlock told Jason Zweig of The Wall Street Journal, most people “are too quick to make up their minds and too slow to change them.”

Most importantly, perhaps, Tetlock encourages us to hunt and to keep hunting for evidence and reasons that might contradict our views and to change our minds as often and as readily as the evidence suggests. One “superforecaster” went so far as to write a software program that sorted his sources of news and opinion by ideology, topic and geographic origin, then told him what to read next in order to get the most diverse points of view.

The best forecasters are all curious, humble, self-critical, give weight to multiple perspectives and feel free to change their minds often. In other words, they are not (using Isaiah Berlin’s iconic description, harkening back to Archilochus), “hedgehogs,” who explain the world in terms of one big unified theory, but rather “foxes,” who, Tetlock explains, “are skeptical of grand schemes” and “diffident about their own forecasting prowess.”

But as Tim Richards has argued, we are both by design and by culture inclined to be anything but humble in our approach to investing. We invest with a certainty that we’ve picked winners and sell in the certainty that we can reinvest our capital to make more money elsewhere. We are usually wrong, often spectacularly wrong. These tendencies come from hardwired biases and also from emotional responses to our circumstances. But they also arise out of cultural requirements to show ourselves to be confident and decisive. Even though we should, we rarely reward those who show caution and humility in the face of uncertainty.

One forecast we should invoke, at least until the evidence demonstrates otherwise, is that the markets will generally trend upward. According to University of Oregon economist Tim Duy, “As long as people have babies, capital depreciates, technology evolves, and tastes and preferences change, there is a powerful underlying impetus for growth that is almost certain to reveal itself in any reasonably well-managed economy.”

Because of the many problems we have with forecasting generally, our portfolios should also be diversified. A diverse portfolio — one that reaches across market sectors — greatly increases the odds that at least some of a portfolio’s investments will be in the market’s stronger sectors at any given time — regardless of what’s hot and what’s not and irrespective of the economic climate.

At the same time, a diverse portfolio will never be fully invested in the year’s losers. For example, according to Morningstar Direct, about 25% of U.S. listed stocks lost at least 75% of their value in 2008 but only four of over 6,600 open-ended mutual funds lost more than 75% of their value that year. Thus a diversified approach provides much smoother returns over time (even if not as smooth as desired!). On the other hand, a well-diversified portfolio will always include some poor performers, and that’s hard for us to abide.

Next, make sure your time horizon is long enough. If you don’t have at least a five-year time frame before using the money, stocks are almost surely a bad idea. That’s because the chances of negative returns over shorter time periods are too high. But over the longer term, our investment prospects are reasonably bright.

Finally, recognize that there are limits to how precise even good forecasting techniques can be. No forecast should be seen as more than the roughest of outlines. Your mileage can and will vary. Keep your attention focused squarely on specific needs, goals and what you can actually expect to control about your portfolio and its results.

Forecasting is a necessary element of financial planning and investment management. Yet we tend to be really bad at it. Thus handle your forecasts and your forecasting process with extreme care.