One of the main signs that investors remain skeptical about the so-far-anemic economic recovery is that plenty of money remains on the sidelines, sitting in cash instruments or under virtual mattresses. A new survey suggests, moreover, that the skittishness of the American investor caused in particular by the markets’ swoon in 2008 may have quite long lasting effects among investors of all kinds, even higher-net-worth ones.

The “Americans’ Investing Outlook Post-Financial Meltdown” survey, conducted in late August on behalf of the international consulting firm AlixPartners, found that 49% of those who identified themselves as “previous investors” said they had either stopped or reduced investing in stocks or mutual funds. A further 26% of those surveyed reported they “had no intention of investing” in either stocks or mutual funds in the next three years, while 27% said they were unsure whether they will invest over the next few years.

Those feelings were stronger among higher-income people, and the intention not to invest was greater among women than among men. For those who reported having annual incomes of more than $75,000, 21% of previous investors reported having stopped investing altogether in stocks or mutual funds. As for gender differences, 32% of female investors said they were more likely not to invest over the next three years, compared to 21% of men.

Clarence Hahn, AlixPartners’ Financial Services practice co-lead, said in a statement that “Investors who had placed their trust in the investment industry are cross, cautious, and confused,” and as a result, advisory firms have two major challenges: “to figure out who really is going to start investing again; and to win back trust by building into their offerings a level of oversight, due diligence, and risk management that will eradicate the possibility of similar meltdowns in the future.”