Do your homework before using alternative investments

The alternative investments industry has doubled in size since 2005, thanks largely to the proliferation of new products — namely, mutual funds — that have made it easier and more cost-effective for investors to gain access to the space.

This is a positive development, as it has allowed more investors to reap benefits previously reserved for institutions and high-net-worth investors, such as increased diversification and better risk-adjusted returns.

Alternatives can be confusing and complex, however. It's for that reason that investors looking to get into the space had better be prepared to do their homework first, according to industry experts.

Ed Butowsky, managing partner of Chapwood Investments, believes alternatives should be a part of all investor portfolios — as much as 30 percent to 35 percent.

"No matter how big your portfolio is, you need to invest in alternatives in order to have a non-correlation to the market," he said. "If you don't have alternatives, you don't have the appropriate risk metrics."

Read MoreDoing well by doing good

Dick Pfister, CEO and founder of AlphaCore Capital, agrees alternatives should have a place with all investors, especially now that there are funds that offer daily liquidity and are registered with the Securities and Exchange Commission.

Following the 2008 financial crisis, he said, investors also seem to have a better understanding of the need for alternatives.

Many people, especially baby boomers, realized they don't have time to weather another 50 percent loss. So, Pfister said, "more and more people have been turning to the alternative world to find a solution."

If used properly, alternatives can offer true portfolio diversification and superior risk-adjusted returns over time. However, the problem is that most investors don't understand them.

Read MoreIs active or passive investing best?

In a 2014 study from life insurance firm Jackson (click here to download), 574 investors were asked to choose a definition of "alternative investments" from a list, and 42 percent selected "I honestly have no idea what alternative investments means." With alternatives, this can be particularly dangerous.

"Like everything in the world of investing, you need to look before you leap — this is especially true in the world of alternatives," Pfister said, adding that there is a huge dispersion between good and bad in the alternatives world.

"There may be a 1 percent difference in returns between large-cap value funds in any given year," he said. "With an alternative fund, the best manager may be up 10 percent and the worst, down 10 percent."

There are various categories of alternatives, so the first step for investors is to know what they're looking for. Research firm Morningstar divides alternatives into seven fund types:

• Bear market: Looks to profit when the market goes down by focusing on short stock positions.
• Managed futures: Uses derivatives such as futures and options to make bets on the direction of markets, such as commodities or currencies.
• Market neutral: Seek to reduce systematic risk using long- and short-selling to offset one another.
• Multicurrency: Invests in various multiple currencies using short-term money-market instruments and derivatives.
• Long-short equity: Invests in a combination of long and short stock positions and related derivatives.
• Non-traditional bond: Looks to deviate in some way from conventional practice in the bond-fund universe, often using unconstrained or absolute-return strategies.
• Multi-alternative: Uses a combination of alternative strategies.

(Illiquid alternative investments, including private equity and venture capital, don't generally fit into an open-end mutual fund structure and are usually considered inappropriate for mainstream investors.)

Morningstar's senior alternatives analyst Josh Charney, in a video segment, "What Good Alternatives Funds Should Do for You," said a main factor investors should consider with alternatives is their correlation to the market and other investments in an investor's portfolio.

"Don't just invest in alternatives to lower a portfolio's beta," he said. "You can lower a portfolio's beta very simply by just selling some securities and moving into cash." An alternative fund should offer some sort of unique characteristic to it — that generally relates to lower portfolio correlations.

Read MoreWill bear market push active management?

Alternatives should also have a low correlation with your other investments. For instance, an investor who has 80 percent in stocks may look to invest the remaining 20 percent in a nontraditional bond funds to achieve more diversification than traditional bond funds. However, some nontraditional bond funds correlate heavily to equities, so that strategy wouldn't make sense.

Investors should certainly be mindful of performance, though comparing returns to the S&P 500 Index is a mistake many investors make; with alternatives, it's more about the Sharpe ratio — a measure of risk-adjusted returns.

"Alternatives should improve the risk-adjusted returns of your portfolio," said Charney. "Long-term the market has a Sharpe ratio of roughly 0.4.

"So you'd want to look for alternatives that have at least that high of a Sharpe ratio, maybe even slightly higher," he added.

Read MoreHelp in making fund choices

Performance is also a frequent source of confusion for investors when their alternative investments underperform the S&P 500 during up markets.

"The way alternatives typically make money is by losing less during market pullbacks," Pfister at AlphaCore Capital said. "With a long-short strategy, you should underperform in an up market.

"If the market is up 20 percent and a long-short [strategy] is up 12 percent, that's probably in line with what they should be doing," he added. "If up 20 percent, then it's probably long-only anyway."

Most alternative investments are intended to outperform over a long time horizon, so they're often not a good fit for investors looking to turn a quick profit. Investors should also be aware of fees, comparing them to other investments in the same peer group — liquidity, transparency and the manager's track record. Chapwood's Butowsky advises "sticking with very large managers that have been around a long time and are institutional-quality."

—By Jennifer Woods, special to CNBC.com