Hedge funds hope to squeeze more gains out of the real estate market despite increasingly muted returns.
Mortgage backed security-focused firms like Structured Portfolio Management, Metacapital Management and Candlewood Investment Group performed well in the third quarter, a snapback from rare losses during the second quarter. And despite lower expectations and even losses at some funds, investors added $1.68 billion to MBS funds over the third quarter, bringing the total of net inflows to the strategy to $5.10 billion for the year, according to hedge fund database eVestment.
"The outlook, in our opinion, for structured credit continues to be a positive one," David Sherr's One William Street Capital Management said in a recent letter to clients which noted opportunities in RMBS, CMBS and other non-housing asset-backed securities. After gaining 18.96 percent in 2012, the fund is up just 4 percent through September.
"We continue to expect increased volatility over the medium term especially if the Fed is in some kind of transition, and we do expect that we will have better opportunities to add exposures back at more attractive levels," the $3.06 billion firm added.
That cautious optimism from One William Street isn't nearly as bullish as MBS funds were a few years ago.
Big money managers were practically jumping up and down about the opportunity after the financial crisis caused MBS prices to collapse. Pensions, endowments, wealthy families and other investors believed the pitch and poured billions of dollars into so-called structured credit funds focusing on residential and commercial MBS.
The opportunity was real. Many hedge funds earned double digit returns from 2009 to 2012 as the American real estate market rebounded and MBS was the top performer across all investment strategies last year. Among the big winners was Brazilian manager BTG Pactual's Distressed Mortgage fund, which gained 45.95 percent net of fees in 2012. Another was former Goldman Sachs trader Josh Birnbaum's Tilden Park Capital Management, which gained 41.35 in its flagship credit fund. And Lehman Brothers veteran Deepak Narula guided his Metacapital Mortgage Opportunities Master fund to a 40.91 percent gain.
This year, the strategy has been subdued. The Absolute Return Mortgage Backed Securities Index, which tracks hedge funds that focus on MBS, is up just 5.52 percent through September. The index had gained 21.95 percent in 2009, 11.53 percent in 2010, 2.34 percent in 2011 and 15.15 percent in 2012.
"The beta trade in RMBS is over, but it's still attractive on a risk adjusted basis relative to other credit securities like high yield," said Carl Ludwigson, head of credit and hedge funds at $7.3 billion wealth advisory firm Bel Air Investment Advisors. "Our expectations are for high single or low double digit returns this year depending on leverage. We weren't expecting the big returns of last year. MBS is still reasonably attractive in a tricky market."
(Read more: Mortgage bond pioneer backing rental bonds)
Those with double digit returns this year include the Canyon Capital Advisors' Canyon Structured Assets Fund (up 17.4 percent through September); the Tilden Park Offshore Investment Fund (up 16.51 percent); Stu Lippman's TIG Advisors' Securitized Asset Fund (up an estimated 15.46 percent); the Credit Suisse Securitized Products Master Fund (up 13.1 percent); BTG Pactual Distressed Mortgage Fund (up 12.91 percent); and the 400 Capital Credit Opportunities Fund (up 10.84 percent).
But other funds have faltered, particularly those that focus on government agency-backed MBS.
Many managers are still recovering from losses in the spring and early summer when interest rates rose on speculation about a decrease in the Federal Reserve's economic stimulus program—which includes the purchase of tens of billions of dollars of agency MBS. Some securities were also hurt by speculation that the government would add to mortgage assistance programs, essentially reducing the amount homeowners owed.
Pine River Capital Management's Liquid Mortgage Fund is down 0.48 percent through September after falling 2.3 percent in May and 1.72 percent in June. The fund, managed by Steve Kuhn and Jiayi Chen, had gained 28.99 percent in 2012.
Don Brownstein's SPM MBS Agency fund is down 14.78 percent through September after gaining 12.05 percent in 2012 and 132.61 percent in 2009. The fund bled money earlier this year, falling 6.49 percent in April, 9.51 percent in May and 0.35 percent in June.
And the Metacapital Mortgage Opportunities Fund is down 4.32 percent in 2013 through last month after losing money each month from March through June, including a decline of 7.26 percent in May.
Still, founder Narula is optimistic on agency MBS.
"A delay in the taper due to all the uncertainty about the debt ceiling and the impact of the government shutdown probably means that these trades have continued upside," he said about agency securities in an October letter to investors.
Narula is also bullish on other types of MBS. "RMBS and CMBS stabilized during the third quarter after weakening earlier in the summer on fears of Fed tapering and higher rates in general," he wrote in the $1.26 billion firm's letter. "Fundamentals remain favorable for both RMBS and CMBS despite the increase in financing costs."
(Read more: There is still opportunity in mortgages: Narula)
Others share that view.
"Agency MBS have rallied significantly and are now rich relative to like-duration Treasuries," bond mutual fund manager PIMCO wrote in a recent PIMCO Mortgage-Backed Securities Fund client note. "In mortgage credit, PIMCO prefers legacy non-Agency MBS, as commercial MBS valuations are largely fair. We continue to look for securities with additional upside from home price appreciation or rep and warranty settlements."
Other observers believe the best days of the RMBS trade are over.
"There's just not a lot of undervalued securities for these mortgage guys to buy anymore. The shelves have been stripped bare—prices are up and there's less opportunity even in the subprime MBS available to buy and fewer auctions by banks and the government," Brian Shapiro, president of risk management consulting firm Simplify, said. "That means funds need to push even further down the rating scale, meaning more risk. And that's not to mention the recent uncertainty around interest rates as result of poorly communicated fed policy,"
Added Shapiro: "Bottom line, the market's moved. Delinquencies are past peak, and the supply of MBS from 2005, 2006, and 2007 Prime and Alt-A paper has shriveled to the size of grapenut."
Even if hedge funds can indeed extract more returns out of MBS, investors are likely to face big price swings to get there.
"We expect volatility in the markets to pick up as we look forward," said $1.42 billion hedge fund firm TIG in an October letter to investors. "Between the leadership transition at the Federal Reserve, the lingering question of when the Fed will actually taper, the tenuous state of the economic recovery and the significant doubts about the ability of Congress and the President to work together to govern, there are many reasons to believe that we should be prepared for a wild ride into the end of the year."
—By CNBC's Lawrence Delevingne. Follow him on Twitter @ldelevingne.