Low cost is just the first step in picking funds: Bogle

What is the strongest predictor of mutual fund or ETF performance?

"Everyone knows cost is the great differentiator," said Jack Bogle, Vanguard Group founder.

The research on fees has been out there for a long time. Bogle has been the leading purveyor of it. But there's a less-covered area of research in which Bogle has been active that's just as interesting. Sit down for this one: Bogle's research shows that fees aren't everything. His research even includes Bogle recommending some active managers.

Bogle does a regularly updated analysis that looks at the correlation between quality (performance over time) and the average expense ratio of the funds in the 50 top mutual fund families. From that analysis, Bogle comes up with a list of the top mutual fund companies for creating high-quality funds. In the past decade, based on Bogle's analyses, the list includes Loomis Sayles, T Rowe Price, USAA, TIAA-CREF and Vanguard.

Jack Bogle
Peter Foley | Bloomberg | Getty Images
Jack Bogle

What's interesting about that list: Three of the five—Loomis Sayles, T. Rowe Price and USAA—are known more for active management than index investing. Many investors might not realize that while Bogle built Vanguard on index funds and is the commander in chief for the index fund revolution, he has long held the door open for some investments in actively managed funds or in active approaches to portfolio strategy.

Bogle wrote in "Little Book of Common Sense Investing" that "there are perfectly reasonable alternative strategies for supplementing the index funds."

Bogle's look at the universe of largest fund companies means that some of the fund families he ranks highly will be those known for active management—most fund companies are active managers.

But the research exercise also makes an important point: Fund expense ratios matter more than any other factor in selecting funds, but that data point alone is not enough to select the right funds for you from the thousands of funds out there.

Here are five keys to fund selection drawn from Bogle's analysis and interviews with him.

1. Buy from a low-fee fund family, but that doesn't mean 'the' low-fee fund or fund family.

The advantage of finding a fund family you're comfortable with versus looking for the lowest-priced fund in each category is that it's a form of one-stop shopping. Basically, you're helping to ensure that when you're constructing the portfolio you're comfortable with, you're fishing from that all-important low-fee pond. "It's the low-cost providers on the top of the list," Bogle said. "We're fairly consistently in the top quartile."

Buying from one fund family is convenient, meaning you'll stick to the discipline of it. Of course, you should always check your family's funds against competitors' funds to make sure you haven't stumbled on an expensive outlier.

Most investors probably don't do this, and over time, adding a high-priced fund or two to your portfolio because it's in the menu offered by your retirement plan, or because someone has recommended it off the cuff, can cost you. More than half of mutual fund-owning households hold funds in a variety of accounts, suggesting a variety of fund families, according to data from the Investment Company Institute.

Read MoreA retirement stress 50-million-plus Americans are facing

Though make sure not to take this to an extreme. Vanguard is beating other companies when it comes to average fees (it was No. 1 overall in Bogle's most recent analysis of the top 50 fund companies), but other factors may weigh in your buying decisions—for good reason.

For instance, USAA, another company with low average fees, has historically been devoted to military members and their families, so it has a variety of services geared toward them. There's also TIAA-CREF, which has grown beyond its roots as an educator's retirement system.

2. Hold for the long term—a fund with a high star rating can burn out quickly.

The real value in low fees comes by compounding over the years. Bogle uses Morningstar's star system to connect performance with low fees, but he emphasized that it's only over time that you can count on the connection. In the short term, a fund with a great return is more likely to revert to average than to keep outperforming.

"The investment field is a very perverse field," Bogle said. "We always look back and think the past is prologue, and that is unequivocally not the case in investing."

Morningstar agrees. Michael Rawson, fund manager research analyst at the Chicago-based company that pioneered the star rating, said it's not the way to pick the next fund to do well in the future. What a star rating does give investors is a risk-adjusted measure of a fund versus its peers—a major improvement over the days when fund companies were choosing whatever benchmark they wanted in order to make their funds look good. The star rating's limitation is why Morningstar also has qualitative analyst ratings.

Read MoreJack Bogle: There's a lot to worry about in the markets

There is one very important feature 5-star funds have in common, and that does matter to long-term performance: low fees. "Low fees do persist," Rawson said. "You shouldn't pick a fund purely by stars, but some things associated with star ratings should be part of the process of selecting funds."

In fact, Morningstar research has found that the chance for success in a cheap fund is twice as great as in a pricey one. Based on 2008 data, for instance, a fund in the cheapest quintile of U.S. equity funds was almost twice as likely to survive and outperform than a fund in the priciest quintile: 56 percent vs. 34 percent.

3. Know there can be fees on top of fees. And don't forget taxes.

Dimensional Fund Advisors also have low average expense ratios, and it's a darling of the registered investment advisor world. But DFA funds are sold only through advisors, so you should inquire about how much the advisor charges on top of any fund fees.

DFA funds may be recommended by independent investment advisors or advisors working for Wall Street firms. They may or may not charge fees that bring your low-priced funds up to the level of high-priced funds. Some companies may use a low expense ratio as a marketing tactic and make up the difference elsewhere in your portfolio.

"In the stock market casino, it is the croupiers who win," Bogle is fond of saying. By croupiers, he means everyone who takes a fee, which could be the fund company but could also be "fund salesmen," such as investment advisors, or brokers who place trades. He also calls the U.S. government a croupier of sorts: It collects taxes.

Because you pay higher taxes on a fund with high turnover—the percentage of stocks in a portfolio that are replaced on an annual basis—investors should look not only at expense ratios but also turnover, Bogle said, adding, "That gives you an additional advantage."

Funds are required to calculate their turnover ratio and publish the information. The Vanguard S&P 500 Fund—the most famous US equity index investment—has a turnover ratio of 2.7 percent. The Fidelity Magellan Fund, an iconic active manager fund—had a turnover ratio of 71 percent, as of March 31, 2015. A turnover ratio of roughly 25 percent implies that the manager is holding stock on average for four years, according to Morningstar.

4. Don't let a fund's fast growth pull a fast one on you.

There is a shift in the market toward low-fee funds. Companies across the board have been dropping their fees. "Everyone tests themselves against Vanguard," Rawson said.

But just because a fund is growing fast doesn't imply that it has a low expense ratio or that it will outperform (remember, those two factors are related). So don't get distracted by the pace of a fund's growth.

In the decade ending in 2014, according to a recent Morningstar report, 95 percent of net inflows went into funds with expense ratios that ranked in the cheapest quintile of their category peers. The asset-weighted expense ratio across all funds was 0.64 percent in 2014, down from 0.65 percent in 2013 and 0.76 percent five years ago.

Yet some of the fastest-growing fund families are also among the most expensive. JPMorgan's market share, for instance, has increased from 1.2 percent to 1.9 percent, according to Morningstar, yet it is middle of the road in terms of both fund quality, as measured by Morningstar, and the average expense ratio.

Read More How are Wall Street-run mutual funds faring? Not so well

He draws a connection between low prices, quality and fund companies' corporate structure. Those that have the highest prices, he said, tend to be financial conglomerates that are serving "two masters"—not only investors but corporate parents demanding high profits and the stockholders the corporations are trying to please. The natural tendency is not to bring expenses down, but to charge ETF and mutual fund buyers as much as they will pay.

"There's a dollar-for-dollar conflict between the two," he said. "[As a fund company], we have a fiduciary duty to investors. But the structure of the industry means that as a business, the job is to increase the business of the management company."

5. Niche funds can make sense in some situations.

Bogle and other advocates for mainstream investors are passionate advocates for broad passive investing, which mirrors the market instead of trying to beat it. But you may enjoy investing—or, if you have a passionate belief in one particular growth trend, like emerging markets investing—you might want to buy index funds in narrower niches for fun, an approach that's somewhere between a broad indexing approach and an active approach.

"A modest holding in a low-cost emerging market index fund is also a reasonable approach, but be sure you understand the risks," Bogle wrote in "The Little Book of Common Sense Investing."

If you do choose to dip your toe in active management, Bogle threw out some guidelines.

Pick active funds run by managers who own their own firms, who follow distinctive philosophies—meaning, who don't merely shadow an index and then charge you more for it—and who invest for the long term. Look at a fund's portfolio holdings (especially the top 10 holdings) on the manager's website or through a fund research site like Morningstar.

"And remember to do your fishing from a low-cost pond," he said in an email to CNBC.

—By Elizabeth MacBride, special to CNBC.com