What's In That Black Box?

Advisors should understand the rocket science fueling the Monte Carlo engine they choose. Jerry Wagner of www.NETirement.com has posted critical comments about some of his competitors on his Web site. Wagner assails Financial Engines for relying on back testing of historical returns to make forecasts. Financeware, when asked to comment on Wagner’s criticism, defends its calculations. But the differences between NETirement’s way of forecasting the future and FW’s highlight why advisors must be able to see through these black boxes.

In taking aim at Financeware, Wagner asks: “If your investment horizon is 30 years, how would you have come out if you started in various years in the past?” Using data from the years 1926 thru 2001, Wagner says you can test 47 possible 30-year runs using Financeware. The first run would be the 30 years from 1926 through 1955, the second run from 1927 thru 1956, all the way through the last 30-year run from 1972 thru 2001. Wagner says this results in a distorted simulation. “The actual returns from 1926 and 2001 each appear in only one run [the first and last],” he says. “Nineteen twenty-seven and 2000 each appear in only two runs [1927 in the first two, and 2000 in the last two]. And 1955 thru 1972 each appear in 30 runs. “Obviously the annual returns are not being equally weighted.”

He adds: “From 1926 through 2001, actual equity total returns have averaged 10.56% annually. By including some years 30 times and others only one time, FinanceWare’s historical audit ended up using an average return of 13.04% [the average return of the 1,410 years [47 tests of 30 years each] over the same period.” Wagner says that a $1000 investment over 30 years with a 10.56% annual return, results in $20,316, while a 13.04% return results in $39,556.