From the June 2014 issue of Research Magazine • Subscribe!

The Limits of Data in Finance and Life

Quantification can be misleading in investment and beyond

© Roy Scott/Ikon Images/Corbis © Roy Scott/Ikon Images/Corbis

For most of her life my mother was a very healthy woman. The only time she ever went to the hospital was to see my dad; and the only reason he was there was because he was the pharmacist. But as she reached her late 80s one malady after another finally landed her in the hospital as a patient.

I still remember my first visit to see her. I went into her room and we chatted for a while—then at some point I excused myself and went out into the hallway to the nurses’ station. I found the nurse who was in charge of my mother and I asked her the obvious question: “How is she doing?”

In my naiveté I expected the nurse to respond something like “Well her strength is better, she's sleeping well, and her appetite is excellent”—the kinds of qualitative signs that the average person would understand. But the nurse didn't say that. Instead she turned around and pulled a clipboard from its plastic holder. Looking intently at the clipboard she began to tell me about my mom's blood pressure, heart rate, oxygenation and other vital signs that to her represented how my mother “was doing.”

This interaction repeated itself every time I visited—and it quickly dawned on me that no one at the hospital had a complete picture of my mother's health.

In addition to my work as an investment advisor, I’m an EMT with a volunteer community emergency rescue service. So I’m more than aware of the importance of vital signs in making a patient assessment—because I measure them regularly. But signs of health are not just those things that lend themselves to quantitative measurement. When I arrive on scene and the patient is an infant, of course we check all normal vital signs. However, the mother's qualitative assessment of her child is a critical element in our evaluation—perhaps the most critical of all.

As I knew intuitively when I visited my mother, quantitative measures, while critically important, tell us only part of the story.

The catalyst for all of this reflection was Zachary Karabell's fascinating new book, The Leading Indicators: A Short History of the Numbers That Rule Our World. Karabell argues convincingly that our most common economic indicators—GDP, CPI, unemployment and trade data—were created for far different purposes and economic circumstances than they are used for today. As a result a growing number of sophisticated observers are discovering that the data from these indicators are deeply flawed and/or completely misleading.

A quick review of the research that fills my email inbox and articles in the financial press suggests that Karabell's insights are not likely to change our reliance on these indicators anytime soon—because flawed or not, as long as enough people believe there is value to them they will continue to be used. This is a classic example of the power and stickiness of simplicity.

One of the fascinating but challenging realities of life is its enormous diversity and almost infinite complexity. A notable benchmark of our advanced civilization is the creation of intellectual tools (or models) that simplify this complexity and allow us to gain both insight into and control over more and more aspects of our life.

Over time these models have become more sophisticated, more reliable and more effective. But like all intellectual tools, in the tradeoff to gain simplicity and utility, they are designed to discount or even ignore huge segments of the real world. This is an important insight: purposeful incompleteness can be enormously valuable—but only as long as the user acknowledges regularly the critical fact that what he is observing is only a partial picture.

Otherwise the temptation is too powerful (especially with very successful models) and the user will slowly begin to believe that the only reality is that which is illuminated by his model.

In the investment world there is a rich history of one flame-out after another that illustrate this dangerous trap: Recent examples include Long Term Capital Management, whose Nobel laureate managers’ unwavering belief in their models nearly brought down the financial system in 1998; or VaR (Value at Risk) a brilliant and successful way Wall Street banks measured the daily risks of their highly leveraged trading positions—until October 2008, when VaR suffered a catastrophic failure and triggered the beginning of what we know as the financial crisis.

The trap doesn't just snare investors: The arrival of GNP (gross national product) in the early 1940s gave us a powerful tool to understand and measure the economy. Prior to the introduction of GNP our understanding of economic output and changes was close to non-existent. GNP became the benchmark by which we measured our wealth and prosperity—but then we got too caught up in the counting: Wealth and prosperity became the primary benchmark by which we valued our lives.

Karabell excerpts a 1968 campaign speech by Robert F. Kennedy that in just a few words captures how an entire society could fall into this trap: “Too much and too long, we seem to have surrendered community excellence and community values in the mere accumulation of material things. Our gross national product … if we should judge America by that, counts air pollution and cigarette advertising and ambulances to clear our highways of carnage…. It measures neither our wit nor our courage; neither our wisdom nor our learning; neither our compassion nor our devotion to country. It measures everything, in short, except that which makes life worthwhile. And it tells us everything about America except why we are proud to be Americans.”

Indicators Awry

Karabell doesn't dwell too long on these unfortunate but recurring examples of the imperfections in our human condition. He wants to discuss a dangerous side effect of over-reliance on useful but ultimately flawed economic indicators; specifically the ones that create “a wide gap between the world we think we are living in and the world in which we live.”

His poster child for this is the trade balance with China. He argues that the conventional wisdom of a massive and extended trade imbalance may actually be a fantasy, driven by an indicator whose fundamental assumptions about world trade haven't been relevant for over 30 years.

Using the iPhone as his primary illustration, Karabell explains the complex reality and immense difficulty of assigning value to its design and manufacturing. With parts coming from at least 12 countries—the final assembly in China (by a Taiwanese owned contract manufacturer) is actually the least important part—worth perhaps $10 per phone to the Chinese economy. But the trade figures follow specific international “rules of origin” protocols that assign $275 per phone to this final phase of manufacture—27 times the actual economic value that actually ends up in China.

All of our decisions are driven by partial information; we just can't know everything. As objective as we try to be, relying too heavily on any one tool, however useful, can actually separate us from the very reality we think we’re measuring. The one thing that can offset this potential and keep us firmly in the real world is the inclusion of our imperfect, behaviorally biased, subjective but common sense observations.

After three days of frustration, we finally hired a private duty nurse to be with my mother for 12 hours a day. With someone available to observe her regularly, the combination of objective and subjective evaluations led to a better understanding of “how she was doing.” That led to a notable improvement in the quality of her care—and ultimately, a much earlier release from the hospital. That was one lesson I knew I’d never forget.

Page 1 of 3
Single page view Reprints Discuss this story
This is where the comments go.