The question remains, which shares should investors buy exactly? Buffett says an index fund is a way to avoid the risk of picking individual stocks.
"The trick is not to pick the right company, the trick is to essentially buy all the big companies through the S&P 500 and to do it consistently and to do it in a very, very low cost way," he added.
Buffett points to the fee savings built into low-cost index funds. The largest such S&P 500 fund, Vanguard's 500 Index Fund, boasts expense ratios of less than a percentage point.
"Costs really matter in investments," said Buffett, who in the past has taken aim at costly funds. "If returns are going to be seven or eight percent and you're paying one percent for fees that makes an enormous difference in how much money you're going to have in retirement."
He added that investors can keep more of their retirement savings by cutting investment costs, by reducing management fees or commissions charged by financial advisors.
"Just remember, the person you're talking to, your fees are their income," the billionaire said.
"And it leaves your pocket and goes to them and you'd better get something for it. And you really don't get it in investment management," he said. "The record shows that the unmanaged index fund is going to do quite well over time and active investment as a group can't beat it."
Most employer-run 401(k) retirement plans offer multiple mutual funds with different assets strategies, but Buffett warned against going with those options, saying "you'll do very well with an S&P index."
He added: "And don't let them talk to you about other index funds…and say you know, you have to have foreign or maybe you can pick a better industry. You'll almost always find it comes with bigger fees."
He conceded that luck can play a role. "Sure, some are going to have better than average just but the fact that if you flip a coin, some are going to call heads and some are going to call tails. And one or the other will be right at that time."
Still, the Oracle of Omaha warned that "you do not want to ever get the impression that you can pick stocks" and that "can enable you to have an edge. It just doesn't work that way."
On the Money airs on CNBC Saturdays at 5:30 a.m. ET, or check listings for air times in local markets.