Jitters over the outlook for the bond market contributed to an uptick in the amount of money flowing into alternative mutual funds—more than $40 billion in 2013. Big winners in the move were nontraditional bond funds, which have wide discretion to invest in a multitude of fixed-income securities (also referred to as unconstrained bond funds).
That's a good example of what can be one of the worst mistakes investors make: panicking when times get tough and abandoning their asset allocation plans. Bond investors did just that at the prospect of rising interest rates. When rates rise, bond prices fall and vice versa. More than a quarter of the 301 advisors who participated in a recent Morningstar survey cited a poor bond market outlook as a key reason to invest in alternatives.
The 2013 flood into alternatives did halve itself this year: Through the end of November, roughly $19 billion had flowed into alternative mutual funds, according to Morningstar. But it's a mutual fund asset class that now has a total of $162 billion in assets from mom-and-pop investors. Whether it's the bond market fears or equities, many investors and financial advisors have latched on to alternatives in recent years.
In theory, alternative investments, within a diversified portfolio, help to take the sting out of market corrections because they behave differently than traditional stocks and bonds. In the aftermath of the financial crisis, investors pumped billions into alternatives in a push to reduce their exposure to stocks while avoiding hunkering down in cash.
"Alternative mutual funds are marketed as funds that miss downturns but capture a lot of upside," said Jason Kephart, an alternative fund analyst at Morningstar. "That sounds great in theory."
Despite the perceived benefits of alternatives, investors ought to proceed with caution, he added.
Read MoreDrunk on liquid alternatives?
"The misconception that has been pushed by fund marketers is that you need to have alternatives," Kephart said. "People shouldn't be rushing into alternatives just to say they have them in their portfolios."
Indeed, it remains to be seen how well the vast majority of alternative funds hold up during a major market correction. There are now nearly 500 alternative mutual funds, and only a handful of them were in existence prior to 2008, according to Kephart.
One of the biggest drawbacks to many alternatives is their mind-bending complexity, said Tim Maurer, director of personal finance for Buckingham and the BAM Alliance. Alternatives are loosely defined as investment vehicles other than stocks, bonds and cash equivalents or funds that hold these three traditional asset types.
Read MoreRisk-wary advisors opt for options
They include things such as hedge funds and private equity funds, which have hefty minimums and are typically sold only to institutions and the ultra-wealthy. Commodities, gold, real estate investment trusts, master limited partnerships, managed futures and certain types of annuities are also alternatives.
Some alternative funds use complex trading strategies to invest in stocks and bonds. Managers of long-short equity funds, for instance, buy stocks (long) that they expect to rise in value and short stocks they expect to drop in value.
Alternatives are going to be one of the first things dumped when they start acting out.Tim Maurerdirector of personal finance for Buckingham and the BAM Alliance
Because of their complexity, many investors don't fully understand alternatives and thus are likely to head for the exits at the first sign of trouble, said Maurer, who attributes the popularity of some alternatives to the "quest for yield." The prolonged low-rate environment in recent years has led many fixed-income investors to search for higher yields than they can get on government and investment-grade corporate bonds.
"The less understanding an investor has when it comes to a given tool, the greater the chances of an investor abandoning that tool," he said. "Alternatives are going to be one of the first things dumped when they start acting out."
Read MoreAlternatives add diversity, cut risk
Liquidity and fees are also major considerations when it comes to alternatives. Some types of alternatives, such as hedge funds, non-exchange-traded REITs and certain annuities are relatively illiquid. Ideally, investors earn a "liquidity premium" for locking up their money for long periods of time in such products, but that premium is not always significant, Kephart said.
Alternative mutual funds have much lower fees than do hedge funds and many other types of illiquid products. Yet alternative mutual funds have significantly higher fees than do traditional mutual funds, and those fees diminish their returns, according to Kephart.
Ironically, alternatives actually have become more correlated to the stock and bond markets since the financial crisis, said David Cowles, a certified financial planner and director of investments at Mosaic Financial Partners. One reason for that may be the proliferation of alternative mutual and exchange-traded funds. The large number of liquid alternative products on the market has made it easier to trade in and out of various asset classes, but the funds are also vulnerable to broad market selloffs, said Cowles.
Read More3 reasons to avoid ETFs
Mosaic allocates about 30 percent of client portfolios to alternatives, up from roughly 10 percent a decade ago. The firm uses passive, low-cost mutual and exchange traded funds to gain exposure to alternatives. Advisors who participated in Morningstar's recent survey allocated 5 percent to 15 percent to alternatives.
"You are still getting a benefit in terms of portfolio diversification with alternatives, but you are getting less of a benefit than you would have gotten before the financial crisis," Cowles said.
