From the June 2002 issue of Investment Advisor • Subscribe!

What's In That Black Box?

Advisors should understand the rocket science fueling the Monte Carlo engine they choose. Jerry Wagner of www.NETirement.com has posted critical comments about some of his competitors on his Web site. Wagner assails Financial Engines for relying on back testing of historical returns to make forecasts. Financeware, when asked to comment on Wagner's criticism, defends its calculations. But the differences between NETirement's way of forecasting the future and FW's highlight why advisors must be able to see through these black boxes.

In taking aim at Financeware, Wagner asks: "If your investment horizon is 30 years, how would you have come out if you started in various years in the past?" Using data from the years 1926 thru 2001, Wagner says you can test 47 possible 30-year runs using Financeware. The first run would be the 30 years from 1926 through 1955, the second run from 1927 thru 1956, all the way through the last 30-year run from 1972 thru 2001. Wagner says this results in a distorted simulation. "The actual returns from 1926 and 2001 each appear in only one run [the first and last]," he says. "Nineteen twenty-seven and 2000 each appear in only two runs [1927 in the first two, and 2000 in the last two]. And 1955 thru 1972 each appear in 30 runs. "Obviously the annual returns are not being equally weighted."

He adds: "From 1926 through 2001, actual equity total returns have averaged 10.56% annually. By including some years 30 times and others only one time, FinanceWare's historical audit ended up using an average return of 13.04% [the average return of the 1,410 years [47 tests of 30 years each] over the same period." Wagner says that a $1000 investment over 30 years with a 10.56% annual return, results in $20,316, while a 13.04% return results in $39,556.

Financeware's Chairman David Loeper concedes Wagner is right on the numbers but says his analysis ignores important issues. "We understand that returns in the center are more heavily weighted," says Loeper. "Historical audit is just one of five simulation methods we offer." He adds that the back-testing calculator provides a simpler way of understanding probabilities, and says Financeware tells advisors that it has a far smaller distribution set of results but that many advisors like to look at a simpler perspective of "the best and worst the market has ever produced."

Says Loeper: "While statistically we concur with [Wagner's] analysis, we like the fact that as luck would have it, our historical audit underweights the two most extreme environments: the 1929 crash and the excessive bull market of the 1980s and 1990s."

Reprints Discuss this story
This is where the comments go.