From the November 2008 issue of Wealth Manager Web • Subscribe!

WHAT FUND RATINGS REALLY TELL US

Morningstar's fund ratings are enormously powerful. Numerous studies have documented that highly rated funds attract more assets. Fund managers face constant pressure to achieve success relative to their peer groups, which is the key to earning four- and five-star ratings and the increased fee income that accompanies a larger asset base.

But what do these ratings really tell us about the funds and their future performance? For advisors and investors, the real value--and in fact the only value--of ratings lies in their predictive ability. If ratings can foretell future risk-adjusted performance, then they can and should legitimately be incorporated into the investment decision process. If not, they are little more than a marketing ploy.

A little bit of history is in order. Morningstar's ratings have been around a long time, and were once the object of justifiable criticism. As a result, Morningstar overhauled its ratings methodology, and released a new version in 2002. With almost six years of historical data for the new methodology, researchers are now able to analyze its predictive value.

The most comprehensive academic analysis was done by two professors at Pace University, Matthew Morey and Aron Gottesman, and was published in 2006. Morey and Gottesman measured the alphas for funds at each rating level, and found that five-star funds had higher alphas than four-star funds, four-star funds were higher than three-star funds, etc. Each fund rating outperformed those at the next lower level. For their analysis, they looked at three years of data from 2002 to 2005.

But their analysis provides no real guidance for advisors. For advisors to achieve the alpha advantage offered by higher rated funds, they would need to buy a diversified portfolio of these funds, which is not practical. Advisors need to know the probability that a fund with a given rating will outperform a fund with a different rating. In 2007, Advisor Perspectives and Morningstar collaborated to answer this question. The results are summarized in the table accompanying this article.

For example, the table shows that a five-star fund had a 54.64% chance of outperforming a four-star fund. Like Morey and Gottesman, we used data for a three year period, but chose to use 2004-2006, since that data was more current. Consistent with Morey and Gottesman, we found that funds with higher ratings outperform those with lower ratings in all cases. The key is that these results provide actionable guidance, in the form of probabilities which can be used in fund selection.

The table shows results for U.S. equities. For other asset classes, including balanced funds and fixed-income funds, the results show an even greater degree of predictability in the ratings, and these are available on the Advisor Perspectives Web site (www.advisorperspectives.com).

Before placing your bets based on this analysis, there are a number of caveats to these results. The three-year period in our analysis (as well as in the Morey-Gottesman analysis) was an almost entirely up market. To fully analyze the predictive value of these ratings, they need to be examined over a complete market cycle, including both bull and bear environments. We hope to undertake this analysis with Morningstar later this year.

Moreover, our analysis used "raw" total returns, and did not adjust for risk. If higher-rated funds are inherently riskier, this could explain their performance advantage. We also did not take into account survivorship bias. Approximately 20% of the funds at the beginning of our three-year period were merged or terminated, and did not exist at the end of the three-year period. These non-survivors tend to be the lower-rated funds--that is the price they pay for not earning a higher rating. Thus, these probabilities most likely understate the true probability of outperformance.

But the biggest caveat with this analysis concerns the peer group approach used by Morningstar in fund classifications, which underlies its ratings methodology. A number of academic studies have shown that peer group classifications are at best, imperfect, and at worst, widely flawed. Fund companies may gain by having their funds mis-categorized, thereby facilitating comparison to a peer group that is easier to beat. Industry consultant Run Surz calls this problem "classification bias"--a funds' holdings are never a perfect match to the universe of securities in its peer group. For example, a mid-cap value fund is likely to hold some securities that are not mid-cap or not value. These biases weaken the results of any analysis based on peer groups.

Advisors should use Morningstar and similar fund ratings with caution. The research shows higher rated funds have outperformed lower rated funds over a limited market cycle. But that research also carries some hefty caveats. In particular, the issue of peer group biases is not well-understood and can easily undermine otherwise rigorous and thorough analyses.

Robert Huebscher, rhuebscher@advisorperspectives.com, is CEO of Advisor Perspectives (www.advisorperspectives.com), a free online database and weekly newsletter for wealth managers and financial advisors.

Reprints Discuss this story
This is where the comments go.