Four hundred years ago ­Jacob Bernoulli had an insight that changed the world. As a result, today we benefit from wisdom that not only help us understand random events but allows us to exploit them for our benefit. Without the work of Bernoulli and those who followed him, we could not evaluate the efficacy of new drugs, we wouldn’t be able to insure anything, weather reports would be useless and Las Vegas would be a backwater railroad depot in the desert. I don’t think that it’s an overstatement that without the development of probability theory, modern life would be close to impossible.

The really fascinating thing, and the reason I’m writing this column, is that—despite the brilliance of Bernoulli and his followers; despite the benefits we have gained from their work—when investors are confronted with randomness they get so uncomfortable that they are willing to act as if it doesn’t exist. And I don’t mean a few Luddites—for starters how about just about everyone in the investment world? The following imaginary report to shareholders will sound outrageous only because you’ve never seen one, not because it isn’t a reflection of reality.

“Last quarter the ABC Value Fund’s net asset value grew by 17.6%, beating the S&P by over nine percentage points. Your investment team met last week to discuss the details of the quarter. We would have liked to report to you that the reason for the blockbuster quarter was our adept timing, our perceptive research, and our time-tested investment process. We indeed would have liked to say that. But we were unanimous in our agreement that the reason for the exceptional performance was just luck.”

Everyone understands that after all is said and done, performance is the single most important factor in investing—and benchmarking (comparing performance to a relevant index) is the primary tool we use to know we’re getting fair value for our risk. There’s just one problem: While the measurement of performance is accurate, it isn’t a complete description of reality.

In his book The Success Equation, Michael Mauboussin illustrates how the ratio of luck (randomness) to skill impacts the outcomes of various activities: In chess, for instance, the outcomes are almost all the result of skill; and in a lottery the outcomes are all luck. While difficult to quantify, the existence of randomness in investing is well documented, especially over the short term. In explaining how randomness impacts returns, Mauboussin writes: “A good process can lead to a bad outcome… and a bad process can lead to a good outcome.”

Taking this observation to its logical conclusion suggests that if randomness is really part of all investment returns, then performance measurement has limited value as tool for decision-making. Mauboussin offers us the following solution: “Since a good process offers the highest probability of a good outcome over time, the emphasis has to be on process.” This is sensible but it goes right up against our discomfort. Mauboussin may be correct in his analysis, but no one is going to care.

The very idea of randomness goes against our instinctual need to find a cause for every effect; making it close to impossible to identify or accept the influence of random events. Daniel Kahneman was awarded his Nobel Prize in part for demonstrating how pervasive this situation is, and how it misleads us in all kinds of ways. Investors are uniquely disadvantaged in this area; not only do we have to face our own internal obstacles, most of the investment industry ignores randomness like the plague.

Skewed Incentives

In the investment world, it’s all about the results. Despite evidence to the contrary, it is much simpler to pin over-performance or under-performance (the effect) on the investment manager (the cause). But the impact of that decision has perverted incentives throughout the investment world, and led investors and managers alike to operate in an Alice in Wonderland reality.

David Tuckett is a London-based psychoanalyst and academic. Six years ago, he interviewed 52 top investment managers in Europe, North America and Asia and published his findings in a small book called Minding the Markets. His subjects were experienced and successful investors at the top of their game. In aggregate they managed over $500 billion in equities. In the course of his interviews Tuckett discovered an unexpected common denominator among his interviewees: They could not execute the long term strategies they knew would be successful—because the pressure to look exceptional over the short term was just too intense. Whether it was fund investors, institutional clients or their own employers—the managers understood that their continuing employment depended upon them outperforming both long-term and short term: an outcome they knew was impossible to achieve.

The managers’ concerns are real. The dominant reason even sophisticated investors choose to hire an investment manager or buy a mutual fund is performance, especially recent performance. A 2008 study of large pension plans showed that nearly all the 9,000 hiring decisions and half the 870 firing decisions were based on performance. Further studies demonstrated that investors gained no benefit from firing one manager and hiring another. Because randomness is part of all returns, those returns act according the law of reversion to the mean.

If you are flipping coins (a completely random activity) and you start with 10 heads in a row, everyone knows that as long as you keep flipping, at some point the results will revert back towards the ultimate 50/50 average (also known as the mean). Reversion to the mean is really that simple; and furthermore, it can’t even occur unless there is randomness.

The recurring observation of reversion to the mean in investment studies proves the existence of randomness in portfolio returns. That is how we can conclude with reasonable certainty that the reason why a good manager is outperforming is not a reflection of suddenly better skills—and the reason why a good manager is underperforming is not a reflection of suddenly worse skills; it’s just a run of good or bad luck. It is also explains why firing underperforming managers and replacing them with outperforming managers does not produce a better return than if investors just stick it out with their original manager.

Let’s review: It is an indisputable fact that randomness exists in all investment returns. Thanks to probability theory we have a robust and successful tool-set to understand and exploit that randomness for our own benefit. And we can see that performance measurement gives us a useful but incomplete picture of how we are doing; so we need to supplement that data with an evaluation of the process of the investment manager. None of these observations are either new or controversial. Yet benchmarking remains the core paradigm for hiring and firing of funds and managers—and is likely to remain so indefinitely.

Navigating past the perverse incentives that infect the investment world and our own instinctual shortcomings is no simple task. But we can still be confident that we can make good choices if we remind ourselves that every outcome in life is the result of the combination of two basic elements: skill and luck.