One of the biggest issues in rolling over a 401(k) to an IRA is that the investor is moving from a fiduciary (the employer)-sponsored plan to an individual account with a broker. A corporate plan also has economies of scale created across its participants to negotiate lower fees. Individuals opening brokerage accounts also likely face more fees, typical of retail mutual fund sales networks.
"When someone retires with a million dollars in a 401(k), in lots of cases they don't know what to do next, and brokers are selling a solution that sounds great," said David Blanchett, head of retirement research at Morningstar. "As someone retires that is when the account balance is largest, and for record keepers that's the most profitable client."
(Read more: The average man's guide to investing during wartime)
Niels Holch, founding partner of Holch & Erikson and executive director of the Coalition of Mutual Fund Investors, has highlighted the high fees and multiple fee levels in the retail investing industry.
"All same fees I've been complaining about would apply to IRAs—It's important for people to think about where they take a rollover IRA," he said.
The Center for Retirement Research said in a report this year that the biggest difference between 401(k)s and IRAs is the relative lack of investor protection in the IRA structure. Regulators have contended that part of the explanation for the high fees on IRA investments is that most IRA assets are with broker-dealers, and the incentive payment arrangements typical in retail mutual funds—namely, 12b-1 distribution fees—encourage the sale of higher-fee mutual funds. The U.S. Government Accountability Office conducted a study this year that concluded the rollover market is, in the least, confusing to retirement plan participants, if not outright misleading.
Blanchett at Morningstar said that the issue is not as black-and-white as a simple high fee-low fee debate, though. For example, participants in a small company's 401(k) plan may pay more in fees than they would with a comparable individual IRA. In addition, higher fees may be justified for investors wanting a an exceptional level of personalized service with an IRA, whereas 401(k) participants are provided minimal personalized service.
(Read more: Investors confused on bond rates: survey)
There is also a simple solution: Chose a discount broker or fund supermarket IRA provider, such as a Vanguard Group or E-Trade. Blanchett said even a low-cost leader such as Vanguard can't offer an IRA that is going to compete on fees with a 401(k) plan including billions of dollars in assets, but keeping IRA assets with one of the larger, low-cost brokers is often the best option for investors that do not need a high level of personalized service.
Over the long term, market forces may favor more competitive offerings, Blanchett said. It's logical to assume that 401(k) record keepers will look forward to rolling over low-fee 401(k)s into IRAs, but that may not turn out to be best strategy.
While an employee is captive to a company's 401(k) offering, once a rollover IRA enters the discussion, there is no guarantee that the record keeper will retain those assets. In addition, he said, more 401(k) providers are looking for ways to keep retiring employees in their plans, though that is still the exception.
Today, long-term investors—especially baby boomers nearing or entering retirement—are faced with a rollover decision. That's the biggest difference between a 401(k) and IRA: As the name suggests, individuals are on their own. The greater focus on 401(k) fees is a function of the captive nature of these plans—as an employee, you have no control over the plan structure or investments.
"While you're in a 401(k), there is fiduciary between yourself and the plan," Vitagliano said. "With an IRA, you can walk into any financial institution by yourself. The individual has to take the initiative."
As more baby boomers move out of company plans, making a choice can be a blessing and a curse, if broker fees aren't closely scrutinized.
—By Eric Rosenbaum, CNBC.com