Backtesting

Many separately managed account (SMA) managers backtest theoretical results, using their models theoretically against how past markets performed. Since the way managers act is often thought to be a constant, one might think that it is fair to look at how things would have worked in the past theoretically.

Backtesting is highly suspect, however. It may be easy to show how the market performed through historical closing prices (for example, Yahoo Finance has a great S&P calculator), but applying a manager’s technique — the way he or she might have acted, given historical data — is problematic. There is simply too much temptation to fudge the result. It’s impossible to know how a manager might have acted at 10:30 AM on a given day, when the market action might be fiercely different than the day’s closing price. Did I mention fudging? Yes, Virginia, there is almost certainly fudging.  

Backtesting is more valid when applied to mutual funds or variable annuity subaccounts, since both investments have only closing prices and not — hour-by-hour, minute-by-minute and second-by-second — changing prices, like stocks and ETFs. Even so, there is every incentive for the backtester to make a scheme’s results look better. I’m not suggesting that fudging is intentional, but trying to make things look better is part of the human condition, and unless the backtesting is done by a non-affiliated third party, I’m suspicious. (Come to think of it, I’m not sure there is a third-party tester.)

Backtesting is fine if you are a SMA manager and want to find out how you might have done, approximately, over past years. I question, though, whether or not it should be used as a selling tool. One better be sure the person doing the backtesting may be trusted, right? And it’s a good idea to understand how the SMA formula and testing are applied.   

As said, backtesting would be more trustworthy if it was accomplished by a third party (not unlike UL tests electrical components).   Anyone want to start a company?