If it would surprise you to learn that the choice between eating a banana or chocolate in a lab experiment has helped to redefine the way Americans save, then you're behind on the latest thinking in "behavioral finance"—the study of how human psychology impacts investing and markets—and that could cost you on the road to a comfortable retirement.
To pioneers in the 401(k) test lab, like Shlomo Benartzi, UCLA professor and chief behavioral economist at Allianz Global Investors, the effort to outsmart our shortsighted, animal side—the one that tends to make irrational financial decisions, especially when it comes to retirement investing—has gotten off to a good start, but it remains a largely unfinished project.
The leading edge of experimentation with retirement messaging will have critical implications for savers over the next two decades, especially among less-affluent Americans.
"The needle is moving in the right direction," Benartzi said. "We've been following measures adopted in the market, and there is good news, but we're also concerned that there is bad news."
Arguably, the single biggest change made to 401(k) plans based on behavioral finance is auto enrollment—the feature that forces employees to "opt out" of a plan or automatically be enrolled in it when they join a company.
You don't need to be familiar with the scientific literature on what is known as "choice architecture" to understand why it's been a big boost to 401(k) plans—it gets around the passive nature of most individuals when it comes to putting off "tomorrow's decisions" today. The results speak for themselves: 60 percent of large employer 401(k) plans now feature auto enrollment.
Benartzi isn't celebrating, though—rather, he sees a wall ahead.
"Behind the scenes, people are very concerned about reaching the point where 60 percent of [large] 401(k) plans have auto enroll," he said. "It seems like that's roughly where we'll end up. I don't think it will go much higher."
(Read more: Less choice is more when it comes to retirement plans)
Benartzi and other investment psychology tinkerers are in the lab looking for other ways around the retirement challenge, and that can help savers stay ahead of the curve. Here is what they see on the 401(k) frontier, broken down into five key concepts:
(Two academic-sounding terms are at the top of the UCLA professor's list: "digital exclusion" and "information architecture." Never fear: It's not as complicated as it sounds when translated into plain English.)
1. If you think you will sign up for a retirement plan online, think again.
Americans live in a world of screens now—and so do retirement-plan sponsors—but for all the ways digital access is making chores easier, it isn't tempting the 40 percent of Americans lacking auto enrollment to participate at a higher rate in 401(k)s. In fact, it's a hindrance, Benartzi said.
Recent statistics shared with Benartzi by a 401(k) plan advisor showed that almost 100 percent of employees sign plan paperwork when it is done at a retirement seminar, but that figure drops to 35 percent when the employees are sent home with the paperwork and have to mail it back in. More eye-opening, when employees are sent home and encouraged to go to a website to join, only 5 percent sign up.
"We might have hit a problem with digital exclusion," Benartzi said. "People have iPhones and Internet access but when they get home have other things to do on the computer. And lots of these websites are not easy to navigate." He added that in "the next five years, we won't see much more improvement in people signing up for a plan unless we figure out new digital approaches. We will see very little participation of new employees."
(Read more: New ways to get workers hooked on a 401(k))
Benartzi said that while white-collar Google workers can navigate a website even if it is overloaded with information, digital exclusion is a problem that is particularly acute among smaller-plan, blue-collar worker–oriented companies in Middle America.
"We have a critical generation in time now, and we can't say the difference between people with or without retirement plans is just the ability to fight through unfriendly websites," he said.
2. Americans still don't have a clue how painful it is to cash out of a 401(k).
In 2012, 39.4 percent of all American workers participated in an employment-based retirement plan, compared with 39.7 percent in 2011. It may not be a big dip, but participation did head in the wrong direction year over year, and that has people like Benartzi worried.
Nevin Adams, director of education for the Employee Benefit Research Institute, said that it's likely the primary "culprit" in the recent slight dip in participation is more a function of people's employment status (unemployed, underemployed, part-time).
"Auto enrollment won't do anything if you're not eligible for the plan," he said.
Auto enrollment also won't do you any good if you cash out of a plan when leaving an employer.
In a world of part-time jobs and switching jobs, workers end up with lots of opportunities over a career to cash out, and many of those individuals are ones struggling to make it in the current economy. For those who might be tempted to cash out, Benartzi said confusion over the implications of that decision is still a huge problem.
"People do not understand the tax hit and how much future return it is taking away from them, and they don't know how much it hurts, especially if they cash out each time they change jobs," Benartzi said. He knows this for a fact: He sat in on a customer-service phone line at a plan administrator, listening to people call in to cash out accounts. "It's an eye-opener. You see how little they understand and how poor a job we do as an industry to help out."
(Read more: Making sure your nest egg doesn't crack)
EBRI data projects that if this "leakage" in retirement assets was reduced by half, it would equal another trillion dollars put into retirement savings over the next decade. "This is a blind spot we need to address," Benartzi said.
One emerging technology that gives him some confidence: more use of personalized videos, so when people try to cash out accounts, they can see on a video screen their name and account balance and what they are about to lose in value. He pointed to Idomoo's pension fund statement video as an example of what could be done.
The messaging can be structured to make sure individuals understand a few key points:
—The IRS will be owed X amount of money, and you won't get it back.
—Every time you switch jobs, you will have less at retirement, if not nothing at all.
—If you really are desperate, you don't have to cash out the entire plan balance.
Victor Ricciardi, finance professor at Goucher College in Baltimore and co-editor of the book "Investor Behavior: The Psychology of Financial Planning and Investing" with H. Kent Baker, said a simple question can explain his reservations about behavioral finance approaches being too broad-based: "Engagement doesn't take into account the full person. If you encourage someone who has $10,000 in credit card debt to throw money into a retirement account, is that a good policy?"
3. Blanket messaging based on fear and greed are risky ...
There isn't a study on the state of retirement in the U.S. released over the past decade that doesn't include some version of the message that Americans aren't saving enough and the implications are dire.
The impact of all these scare tactics? Not sufficient, said David Musto, CEO of J.P. Morgan Retirement Plan Services.
"We still have low participation rates in 401(k) plans, and the average income-deferral rates, while we've seen some improvement, is still around 6 percent," he said. "Most would agree saving [at least] 10 percent would be better."
When it comes to behavioral finance and decision-making, auto enrollment is just a start, and providing estimates of retirement-balance shortfalls fall far short of the mark in spurring participants to save more.
(Read more: 40-plus is not too late to save for retirement)
"One of the challenges of educating people is [that] the baseline message can be depressing," said Don Hess, head of product development for J.P. Morgan Retirement Plan Services.
The company at one time relied more on blanket and fear-based messaging, telling employees how much they were on track to have in retirement and how much that left them to make up if they wanted to have a secure retirement—the gap was typically large. "That message worked horribly," Musto said, because it shows the company doesn't know their personal life situation. "Who could make up that amount?"
Hess agreed. "A simple question around what the participant will draw down in retirement will have an answer that is radically different, based on age and family history and Social Security and the tax basis of money in retirement."
Now J.P. Morgan Retirement Services uses a less judgmental approach, focusing on the concept of "social norming" or, in plain English, showing your progress relative to your peers and, if they are on track to be successful savers, showing you how they got there.
"We don't say, 'This is what you need'; we say, 'Here is what you are on track to achieve, and if you were to increase by 2 percent, that number would change to y,'" Musto said. "It's not frightening them."
4. ... But fear and greed do still work as a motivator when tailored to the individual.
Nevertheless, if you personalize your message to certain age groups, fear and greed will indeed make an impact, Musto said. In other words, greed is a better message for the 20-year-old, and something closer to fear, if not fear exactly, is a better message for the 50-year old.
"For the 20-year-old ... it is about what you could have if you participate," Musto said. "It's all about potential with a younger participant. But someone in their 50s, it is time to assess; they are ready at that point."
(Read more: Five years later, still traumatized by market)
5. Toward a unified theory of investing (i.e., investors still are not rational beings)
Even if all of this effort to get more workers to enroll in, contribute to and stay longer in 401(k) plans works, Ricciardi sees a big challenge in the concept that underlies long-term investing, known as Modern Portfolio Theory.
MPT, put most simply, is the idea that a diversified portfolio across investment asset classes reduces risk the best over the long term, but that's not an approach that takes into account an individual's risk tolerance, especially after the financial crisis.
Goucher College's Ricciardi said his data shows that if someone was in stocks before the crisis and then shifted all of their money out of stocks, they remain traumatized.
"There is a high percentage of people who formerly had money in the market who now have nothing in stocks," Ricciardi said. "It will take a long time to get those people to change their behavior. If you have a risk-averse person and try to push them into riskier assets, all you make them do is shift them to conservative assets."
He added: "MPT is a good model to teach people the basics, but the underlying assumption is people are rational. If you help people understand their emotional baggage ... that's where it can work. Ultimately, I want to take from both behavioral finance and MPT and make a better overall theory of finance."
Certified financial planner Tim Maurer, director of personal finance for the BAM Alliance, said his "personal" investing theology more and more is organized around one idea: Personal finance is more personal than it is finance.
"It's hard not to marvel at studies that suggest as much as 80 percent of financial and estate-planning recommendations are not implemented," Maurer said. "It's either a tremendous failing on the part of the financial industry—or its clients—or both. My hypothesis regarding the foremost factors playing into the 80 percent conundrum is that financial-planning recommendations aren't aligned enough with the highly individualized values and goals of clients."
(And if you're still wondering how choosing bananas over chocolate relates to retirement saving, Benartzi explains in this video.)
—By Eric Rosenbaum, CNBC.com