Americans are feeling less confident about their financial situation, according to a new survey by Hearts & Wallets. For the second year in a row, according to the survey released Tuesday, investors reported declining levels of financial confidence, with just 13% saying they felt “confident, comfortable and secure” about their current situation. Just one in three said their retirement savings were on track.

Respondents with less than $100,000 in investable assets are driving the increase in financial anxiety, but Hearts & Wallets found all income levels reported an increase in concern.

A big factor in that increase is no doubt the recent announcement that there will be no increase in the cost of living adjustment for Social Security. Concerns about the future of Social Security moved from the No. 3 concern in 2014 to No. 1 this year.

People are more concerned about their ability to save for retirement, but Laura Varas, co-founder and partner at Hearts & Wallets, said that investors still want to feel like they’re in control. Almost two-thirds wish they were doing a better job of saving, especially those with less than $100,000 or Gen X and Gen Y investors.

“People want to be in control of their savings and retirement,” Varas said in a statement announcing the results of the survey, “but the average American is finding it tougher to accumulate retirement savings given distortions in the economy, including low interest rates. They fear market volatility and are frustrated that they must put assets at more risk to earn an acceptable return on their investments.”

“Consumers want government to solve their most pressing concern of the reliability of Social Security,” Chris Brown, co-founder and partner, added. “Beyond that, consumers want to take responsibility for their own financial lives.”

That could be a problem, as almost half of respondents said they were “inexperienced” investors this year, compared with 35% last year, and 41% said they were confused by the amount of information out there.

Despite feeling ill-prepared to invest on their own, only a third see value in working with a financial advisor.

“It’s healthy that some consumers want to pay for advice from financial professionals, while others prefer solutions that embed advice into products, and still others just want the lowest cost solution of indexing,” Varas said.

In addition to these higher levels of anxiety over their broader financial situation, another survey found investors, women in particular, are anxious about their retirement prospects specifically. The Insured Retirement Institute found 80% of women are concerned they’re not saving enough, with 54% saying they’re “very concerned.”

Women are almost twice as likely as men to express doubt over their retirement prospects. Fifty-four percent of women said they were very concerned about being able to retire when they want to, compared with 34% of men; 53% of women said they were very concerned about being able to afford the lifestyle they want in retirement, compared with 36% of men.

Cathy Weatherford, president and CEO of IRI, pointed to familiar headwinds women face — income disparities, time out of the workforce, longer lifespans — that effect their retirement prospects, and urged the financial services industry to build a “more thorough understanding of women’s priorities, values, and preferences.”

For example, women are most concerned about debt, paying bills, and investments declining in value. They’re also less likely to be do-it-yourself investors: 20%, the survey found.

That doesn’t mean they’re asking advisors for help, though. Fifty-seven percent said they turn to friends and family for financial advice and 40% ask co-workers.