"For many people, it's all they have," said Jeanne Thompson, a vice president with Fidelity Investments.
Many Americans also appear to be worried it's not going to be enough. An annual survey of more than 4,000 full-time workers participating in retirement plans, released Wednesday by human resources consulting firm Towers Watson, found that less than one-fourth of respondents are very confident they'll have enough money for the first 15 years of retirement.
About half of the respondents to the Towers Watson survey had reviewed their retirement savings in detail in the past year.
Thompson, of Fidelity, said that for many participants, a retirement plan is also their only exposure to the stock market. And over the years, investment experts have come to realize that most people don't have the will to figure out how to invest their money correctly—and may not have the skill, either.
Read MoreValue of college? For millennials, $17,500 a year
"If you look at the do-it-yourself investor, it's like they're all over the place," said Vanguard's Young. "It looks like some people appear to be getting it right, but we wonder if that's really skill, or is it luck?"
Most people are just leaving well enough alone.
At Vanguard, Young said about 10 percent of participants initiated at least one trade or exchange in 2013. At Fidelity, Thompson also said that about one in 10 investors make a change in a given year.
Consulting firm Aon Hewitt's 401(k) Index, which has tracked investment activity at large U.S. companies since 1997, shows that in a typical year, fewer than 2 in 10 retirement-plan participants move money from one fund to another.
The share of people transferring money didn't even increase that much in 2008, when the markets tumbled, said Winfield Evens, a partner at Aon Hewitt.
"It's rare for people to make a transfer," Evens said.
There's definitely a risk to moving money around at the wrong time. Thompson said Fidelity's analysis showed that the people who panicked and moved their money to cash when the market tanked during the financial crisis often didn't get back into the market in time. That means they lost out compared to those who had stayed the course once the markets started to improve.
Experts say the one area you can—and should—easily pay attention to is how much of your income is going toward retirement. One key worry is that many Americans just won't have enough money in those accounts once they retire to maintain their standard of living.
"People aren't saving enough," Munnell said.