For millions, 401(k) plans have fallen short

You need to know this number: $18,433.

That's the median amount in a 401(k) savings account, according to a recent report by the Employee Benefit Research Institute. Almost 40 percent of employees have less than $10,000, even as the proportion of companies offering alternatives like defined benefit pensions continues to drop.

Older workers do tend to have more savings. At Vanguard, for example, the median for savers aged 55 to 64 in 2013 was $76,381. But even at that level, millions of workers nearing retirement are on track to leave the workforce with savings that do not even approach what they will need for health care, let alone daily living. Not surprisingly, retirement is now Americans' top financial worry, according to a recent Gallup poll.

To be sure, tax-advantaged 401(k) plans have provided a means for millions of retirement savers to build a nest egg. More than three-quarters of employers use such defined contribution plans as the main retirement income plan option for employees, and the vast majority of them offer matching contribution programs, which further enhance employees' ability to accumulate wealth.

But shifting the responsibility for growing retirement income from employers to individuals has proved problematic for many American workers, particularly in the face of wage stagnation and a lack of investment expertise. For them, the grand 401(k) experiment has been a failure.

"In America, when we had disability and defined benefit plans, you actually had an equality of retirement period. Now the rich can retire and workers have to work until they die," said Teresa Ghilarducci, a labor economist at the New School for Social Research who has proposed eliminating the tax breaks for 401(k)s and using the money saved to create government-run retirement plans.

A historical accident?

It wasn't supposed to work out this way.

The 401(k) account came into being quietly, as a clause in the Revenue Act of 1978. The clause said employees could choose to defer some compensation until retirement, and they would not be taxed until that time. (Companies had long offered deferred compensation arrangements, but employers and the IRS had been going back and forth about their tax treatment.)

"401(k)s were never designed as the nation's primary retirement system," said Anthony Webb, a research economist at the Center for Retirement Research. "They came to be that as a historical accident."

History has it that a benefits consultant named Ted Benna realized the provision could be used as a retirement savings vehicle for all employees. In 1981, the IRS clarified that 401(k) plan participants could defer regular wages, not just bonuses, and the plans began to proliferate.

Read MoreHere's one way not to fall short on retirement savings

By 1985, there were 30,000 401(k) plans in existence, and 10 years later that figure topped 200,000. As of 2013, there were 638,000 plans in place with 89 million participants, according to the Investment Company Institute. And assets in defined contribution plans totaled $6.6 trillion as of the third quarter of 2014, $4.5 trillion of which was held in 401(k) plans.

"Nobody thought they were going to take over the world," said Daniel Halperin, a professor at Harvard Law School. who was a senior official at the Treasury Department when 401(k) accounts came into being.

Rise of defined contributions

But a funny thing happened as 401(k) plans began to multiply: defined benefit plans started disappearing. In 1985, the year there were 30,000 401(k) plans, defined benefit plans numbered 170,000, according to the Investment Company Institute. By 2005, there were just 41,000 defined benefit plans–and 417,000 401(k) plans.

The reasons for the shift are complex, but Ghilarducci argued that in the early years, "workers overvalued the promise of a 401(k)" and the prospect of amassing investment wealth, so they accepted the change. Meanwhile, companies found that providing a defined contribution, or DC, plan cost them less. (Ghilarducci studied 700 companies' plans over 17 years and found that when employers allocated a larger share of their pension expenditures to defined contribution plans, their overall spending on pension plans went down.)

But the new plans had two key differences. Participation in 401(k) plans is optional and, while pensions provided lifetime income, 401(k) plans offer no such certainty.

"I'm not saying defined benefit plans are flawless, but they certainly didn't put as much of the risk and responsibility on the individual," said Terrance Odean, a professor of finance at the University of California, Berkeley's Haas School of Business.

Early signs of trouble

That concept may not have been in the forefront of employees' minds at the start, but problems with 401(k)s surfaced early.

For one thing, employee participation in 401(k) plans never became anywhere near universal, despite aggressive marketing by investment firms and exhortations by employers and consumer associations to save more. A 2011 report by the Government Accountability Office found that "the percentage of workers participating in employer-sponsored plans has peaked at about 50 percent of the private sector workforce for most of the past two decades."

The employees who did participate tended to be better paid, since those people could defer income more easily. The GAO report found that most of the people contributing as much as they were allowed tended to have incomes of $126,000 or more.

Read MoreDo you know where your 401(k) money is going?

In part, that is because the ascent of 401(k) plans came as college costs started their steep rise, hitting many employees in their prime earning years. Stagnating middle-class wages also made it hard for people to save.

Fees have been another problem. Webb has studied 401(k) fees, and he concluded that "as a result of high fees, fund balances in defined contribution plans are about 20 percent less than they need otherwise be."

The Department of Labor in 2012 established new rules requiring more disclosure of fees, but it faced strong industry opposition, including a 17-page comment from the Investment Company Institute.

Failure of choice

Most employees also turned out to be less than terrific investors, making mistakes like selling low and buying high or shying away from optimal asset classes at the wrong time.

Berkeley's Odean and others have studied the effect of investment choice on 401(k) savers, and found that when investors choose their asset class allocation, a retirement income shortfall is more likely. If they can also choose their stock investments, the odds of a shortfall rise further.

"401(k)'s changed two things: you could choose not to participate, and you chose your own investments, which a lot of people, I think, screw up," Halperin said.

Read MoreHow many 401(k) investment choices are too many?

Benna, who is often called the father of the 401(k), has argued that many plans offer far too many choices. " If I were starting over from scratch today with what we know, I'd blow up the existing structure and start over," he said in a 2013 interview.

Another problem is that when 401(k) savers retire, they often opt to take their savings in a lump sum and roll the money into IRAs, which may entail higher fees and expose them to conflicted investment advice. A recent report by the Council of Economic Advisors found that savers receiving such advice, which may be suitable for them but not optimal, see investment returns reduced by a full percentage point, on average. Overall, the report found that conflicted investment advice costs savers $17 billion every year.

The result of all these shortcomings? Some 52 percent of American households were at risk of being unable to maintain their standard of living as of 2013, a figure barely changed from a year earlier—even though a strong bull market should have pushed savings higher and the government gives up billions in tax revenue to subsidize the plans.

In a hearing last September on retirement security, Sen. Ron Wyden, D-Ore., declared that "something is out of whack. The American taxpayer delivers $140 billion each year to subsidize retirement accounts, but still millions of Americans nearing retirement have little or nothing saved."

Retirement worries rise

As problems mount with 401(k)s, Americans' worries about retirement security are intensifying.

A 2014 Harris poll found that 74 percent of Americans were worried about having enough income in retirement, and in a survey published recently by the National Institute on Retirement Security, 86 percent of respondents agree that the country is facing a retirement crisis, with that opinion strongest among high earners.

Changes may come, but for now, 401(k) plans and their ilk remain Americans' predominant workplace retirement savings vehicle. They may be a historical accident, but for the millions of people now facing a potentially impoverished retirement, the fallout is grave indeed.

As a former Treasury official, Halperin witnessed the creation of 401(k) accounts, But, "on balance, I don't think it was a big plus" that the accounts were created, he said. "I don't take credit for it. I try to avoid the blame."