Move over, 401(k): IRA fees are what really hurt

Courtney Keating | E+ | Getty Images

You've likely heard the complaint: Retirement plan fees are too high and are cutting into your long-term investment returns. Even if you haven't heard it directly from Jack Bogle—and didn't see the "Frontline" examination in which the Vanguard Group founder was featured lambasting the 401(k) industry for fee-gouging—you've probably been convinced somewhere along the way that 401(k) fees are excessive.

But what about individual retirement accounts (IRAs)? If you don't hear as much about high IRA fees, you should listen up.

Here's one big reason: There is more money in IRAs than in 401(k) plans in the U.S., and that means more opportunity for Americans to give away more of their long-term returns to fees if they aren't careful. At the end of 2012, $5.4 trillion was invested in IRAs, versus $5.1 trillion in defined contribution plans—the plan type that includes 401(k)s, according to the Investment Company Institute.

"IRAs, where 401(k) assets unfortunately end up, have as high if not even higher fees on average," said Mercer Bullard, a law professor at the University of Mississippi School of Law and longtime critic of the fund industry.

(Read more: The US isn't the 'place to be' for investors?)

With the generational transfer of baby boomer wealth out of company-sponsored defined contribution plans and into IRAs as more boomers retire, the IRA asset trend is set to increase over the next half decade.

Francis Vitagliano, research consultant at Boston College's Center for Retirement Research, began studying the retirement industry when the Employee Retirement Income Security Act (ERISA) was made law. At the time, he said, no one thought that most retirement assets would end up in IRAs. Regulators had always contemplated IRAs as being supplemental savings vehicles.

"The market has focused on the growth of 401(k)s and the fees, but what has been kind of overlooked is the IRA rollover market," Vitagliano said. "The same light shined on 401(k) assets and fees should be directed at rollover IRAs."

One of the biggest issues in rolling over a 401(k) to an IRA is that the investor is moving from a fiduciary (the employer)-sponsored plan to an individual account with a broker. A corporate plan also has economies of scale created across its participants to negotiate lower fees. Individuals opening brokerage accounts also likely face more fees, typical of retail mutual fund sales networks.

"When someone retires with a million dollars in a 401(k), in lots of cases they don't know what to do next, and brokers are selling a solution that sounds great," said David Blanchett, head of retirement research at Morningstar. "As someone retires that is when the account balance is largest, and for record keepers that's the most profitable client."

(Read more: The average man's guide to investing during wartime)

Niels Holch, founding partner of Holch & Erikson and executive director of the Coalition of Mutual Fund Investors, has highlighted the high fees and multiple fee levels in the retail investing industry.

"All same fees I've been complaining about would apply to IRAs—It's important for people to think about where they take a rollover IRA," he said.

The Center for Retirement Research said in a report this year that the biggest difference between 401(k)s and IRAs is the relative lack of investor protection in the IRA structure. Regulators have contended that part of the explanation for the high fees on IRA investments is that most IRA assets are with broker-dealers, and the incentive payment arrangements typical in retail mutual funds—namely, 12b-1 distribution fees—encourage the sale of higher-fee mutual funds. The U.S. Government Accountability Office conducted a study this year that concluded the rollover market is, in the least, confusing to retirement plan participants, if not outright misleading.

Blanchett at Morningstar said that the issue is not as black-and-white as a simple high fee-low fee debate, though. For example, participants in a small company's 401(k) plan may pay more in fees than they would with a comparable individual IRA. In addition, higher fees may be justified for investors wanting a an exceptional level of personalized service with an IRA, whereas 401(k) participants are provided minimal personalized service.

(Read more: Investors confused on bond rates: survey)

There is also a simple solution: Chose a discount broker or fund supermarket IRA provider, such as a Vanguard Group or E-Trade. Blanchett said even a low-cost leader such as Vanguard can't offer an IRA that is going to compete on fees with a 401(k) plan including billions of dollars in assets, but keeping IRA assets with one of the larger, low-cost brokers is often the best option for investors that do not need a high level of personalized service.

Over the long term, market forces may favor more competitive offerings, Blanchett said. It's logical to assume that 401(k) record keepers will look forward to rolling over low-fee 401(k)s into IRAs, but that may not turn out to be best strategy.

While an employee is captive to a company's 401(k) offering, once a rollover IRA enters the discussion, there is no guarantee that the record keeper will retain those assets. In addition, he said, more 401(k) providers are looking for ways to keep retiring employees in their plans, though that is still the exception.

Today, long-term investors—especially baby boomers nearing or entering retirement—are faced with a rollover decision. That's the biggest difference between a 401(k) and IRA: As the name suggests, individuals are on their own. The greater focus on 401(k) fees is a function of the captive nature of these plans—as an employee, you have no control over the plan structure or investments.

"While you're in a 401(k), there is fiduciary between yourself and the plan," Vitagliano said. "With an IRA, you can walk into any financial institution by yourself. The individual has to take the initiative."

As more baby boomers move out of company plans, making a choice can be a blessing and a curse, if broker fees aren't closely scrutinized.

—By Eric Rosenbaum, CNBC.com