The underlying trends in Wells Fargo survey are just as distressing, if not surprising.
"What we heard is that paying day-to-day bills trumps planning for retirement," Nordquist said. "Day-to-day bills are really overwhelming them."
Fifty-nine percent said their top financial concern is everyday bills, and 42 percent told Wells that it is not possible to both pay them and save for retirement.
"As much as we read about the economy turning around, people in middle-class America are not feeling it yet," Nordquist said.
(Read more: 6 tips from Jack Bogle on teaching kids to invest)
Even for those with assets to invest for retirement, lack of a plan and lack of confidence in the stock market hold them back from building a nest egg.
Only 30 percent have a retirement plan. The benefit of having a strategy is huge: Wells Fargo found that those with a plan have saved three times as much toward their retirement goal as those without one.
"That's really amazing, and it makes a really big difference, whether your income is $25,000 or $100,000," Nordquist said. "The plan doesn't need to be elaborate."
Yet the survey found that 75 percent of middle-income Americans said they are not confident that stock market is a good place to save for retirement. That finding holds true for both young and old.
(Read more: Why saving for retirement keeps getting harder)
In fact, respondents in their 20s are the least sure, with 80 percent saying they are not confident about investing in the market.
"They have the long runway and should be taking advantage of appreciation," Nordquist said. Yet when Wells Fargo asked what they would do if given $5,000 to invest, most said they would put it in a savings account.
One of the survey's more encouraging findings was that only 7 percent of Americans "strongly agree" that gambling in Las Vegas is as good an idea as investing in the stock market, but that's little solace (and 27 percent of Americans surveyed generally agreed with the statement).
Starting early and having a plan are the most important steps, Nordquist said.
"Those two have a lot more to do with what you end up with than the amount, even if you're saving at a 6 percent to 10 percent level," Nordquist said. "When you're young but not in the market, you won't build the nest egg."
People in their 30s were the most likely to have a retirement plan in place, but 45 percent of respondents said they have so few assets there is no reason for them to have a plan, while another 25 percent said they don't know how to create one.
"The ones without a plan we worry about," Nordquist said.
Their plan could become to work until they drop.
—By Eric Rosenbaum, CNBC.com, and Anthony Volastro, CNBC Segment Producer