In the aftermath of the 2008 financial crisis, shell-shocked investors did a fair amount of soul-searching with regard to stocks. These days, they are working through their angst when it comes to the bond market, which has likely come to the end of a very long bull run.
Last year was one of the worst years for the bond market in a long time, and many investors and advisors are skeptical about its prospects in light of the very strong possibility that interest rates will continue to climb from their rock-bottom lows. At the very least, interest rates, which have an inverse relationship to bond prices, are expected to rise modestly and gradually in the near term.
"The 2008 downturn caused an instant and immediate focus on the stock market. People have generally assimilated that risk and adjusted their portfolios as a result. What many investors haven't been prepared for is the fact that bonds can decline substantially in value as well," said Mark Rosenbaum, president and chief executive officer of Rosenbaum Financial.
(Read more: Buckle up for a likely bond-rate bump)
The uncertain outlook for the bond market has led to a surge in the popularity of alternative investment strategies, once the domain of institutions and high-net-worth investors.
A survey of advisors and institutional investors taken last year by Morningstar Inc., an investment research firm, found that a "poor bond market outlook" is among the top five reasons advisors are allocating a portion (typically 10 percent to 20 percent) of client portfolios to alternatives. Advisors are also driven by a desire to invest in ways that aren't highly correlated to the stock and bond markets.
"For the first time in a long time, in 2013 people saw losses on the bond side of their portfolios, and they started wondering what to do," said Gary Ribe, director of research and investment management at MACRO Consulting Group.
Alternative investment strategies come in a dizzying array of forms and risk profiles. But it is not hedge funds and other illiquid vehicles with hefty minimums that are gaining popularity. Investors of all types have pumped billions of dollars in recent years into alternative-strategy mutual and exchange-traded funds, many of which were rolled out post-financial crisis.
(Read more: Ready to bail on bonds? Not so fast)
Many advisors view alternative funds as surrogates for bonds, ones that should allow clients to earn more than they can make right now in traditional fixed-income investments without taking equity-like risk. That means such funds aren't likely to generate eye-popping returns, but they should help to take the sting out of a major stock and/or bond market correction.
Total assets under management in alternative mutual funds, including nontraditional bond funds, surged to $255 billion last year, more than double the amount for 2010, according to Morningstar. By some estimates, assets in alternative mutual funds will grow to represent 13 percent of all mutual fund assets in a few years, according to a report by Fortigent, which provides research, technology and portfolio management to independent advisors and banks.
Alternative-style funds have much more latitude than traditional funds, boasting the ability to invest long and short across a range of asset classes, from stocks to bonds to currencies and commodities. Some have a single manager with a narrow focus, and others are funds with allocations to multiple managers with different strategies.
Nontraditional bond funds—which generally can invest across the fixed-income spectrum—and long-short equity funds are two of the most popular types. The latter buy stocks (long) that managers expect will climb in value; short equities are anticipated to drop in value. A short sale is a mechanism that allows an investor to profit from the decline in value of a stock over a set period of time.
Last year the Pimco Unconstrained Bond Fund, the largest fund of its type, had nearly $30 billion in assets under management, up from less than $10 billion at its inception in 2008, according to Morningstar.
"In essence, alternative strategies give the fund manager great flexibility to either bet with or against certain securities or markets," explained Barry Glassman, president of Glassman Wealth Services.
"Within our portfolios, at a minimum, alternative strategies should be diversifiers," added Glassman, a certified financial planner. "They shouldn't zig and zag in tandem with the stock or bond markets."
Given the complexity of alternative funds and the fact that many haven't been around very long, advisors say it behooves potential investors to do their homework thoroughly.
Many advisors seek to put together a combination of such funds that aren't highly correlated in terms of performance. Complicating their due diligence efforts is the fact that even funds within the same category often invest in very different ways, said Scott Welch, chief investment officer at Fortigent.
"The fact that retail investors have access to a more diverse set of potential return drivers is a positive thing. You just really need to know what you are buying," Welch explained.
Investors also ought to be aware that alternative funds generally have higher management fees than traditional mutual funds, which is partly why investment management firms have been eager to roll them out in an era of low-cost indexing. The average alternative fund charges between 1.25 percent and 2 percent of assets under management, but some charge as much as 3 percent, according to research from Fortigent. Of course, fees impact investor returns.
(Read more: Alternative investments grow on advisors)
"The biggest concern about alternative mutual funds is that they will underperform from a cost-benefit standpoint," said Josh Charney, an alternative investments analyst at Morningstar.
Dorothy Weaver, chief executive and co-founder of Collins Capital Investments, said potential investors ought to seek out fund managers who have a long track record in the investment strategy they are pursing and also "eat their own cooking."
"If you know they have their money sitting alongside yours, that will drive their behavior," said Weaver, the former chairperson of the Federal Reserve Bank of Miami.
(Read more: The pros and cons of nontraded REITs)
With interest rates likely to rise and the economy steadily improving, even advisors who also invest in less-liquid alternative strategies on behalf of wealthy clients are rethinking some of their options. Rosenbaum Financial, for instance, has invested on behalf of certain clients in private, nontraded real estate investment trusts formed in the wake of the financial crisis.
During the early stages of the economic recovery, many such REITs, which typically pay dividends of 5 percent to 6 percent, took advantage of distress in the real estate market to snap up properties that have since climbed in value. But rising rates are likely to increase their cost of capital, and commercial real estate prices have recovered to a large degree, Rosenbaum said.
"We liked these through the end of last year, but we are looking very cautiously at these REITs this year," he explained.
—By Anna Robaton, Special to CNBC.com