Are commercial mortgages the next big thing for hedge funds?

Gleaming office towers, corporate office parks and big box retail stores could be the next big thing for Wall Street traders.

Some of the smart money certainly thinks so. Hedge funds focused on buying and selling securities backed by assets like home loans and credit card payments—known as structured credit or asset backed funds—are increasingly betting on commercial mortgages.

A small but growing group of money managers believe that they can earn returns between 10 percent and 20 percent annually by trading commercial mortgage backed securities, which are essentially office and retail loans bundled by bankers like the $3.5 billion one JPMorgan Chase and Deutsche Bank just did for Hilton Worldwide.

Those profit expectations are lower than what many hedge funds made betting on residential mortgages after the financial crisis, but the returns are still healthy compared to other types of bonds, like Treasury or high-yield corporate credit which usually yield single-digit percentage returns.

Angelo, Gordon & Co., Cerberus Capital Management and Claros Fund Management have earned double-digit returns on their CMBS investments this year. Other firms, such as Pine River Capital Management, CQS and Ellington Management Group have all seen gains in their credit focused funds as they increased bets on the sector over 2013.

"We see a great opportunity set for CMBS as valuations and issuances have come back in a meaningful way. It's a good-sized opportunity," said Leo Huang, a portfolio manager at Ellington who specializes in CMBS.

(Read more: Hedge funds hope for more mortgage juice)

Lots for traders to love

Investors like CMBS for several reasons.

First, prices are all over the place: Old securities with the highest rating, AAA, have recovered from their post-financial crisis lows in 2008 and 2009.

But lower-rated securities—A and BBB- for example—are still near their nadir after falling steeply in 2007 and 2008. About two thirds of the approximately $810 billion U.S. CMBS market is rated AAA, according to JPMorgan and Bloomberg data. The riskier tranches of CMBS are also volatile, which sometimes scares off more conservative institutional investors.

Second, new securities are finally being packaged and sold again, creating new places to bet. CMBS issuance is about $90 billion this year—nearly double 2012—but still well off from the record high of $288 billion in 2007, according to Dealogic. But some hedge funds believe the credit quality in new CMBS is deteriorating given low interest rates, creating the opportunity to short certain deals.

Third, the commercial market's recovery hasn't been as even as the housing market. American jobs have been slow to come back, for example, and certain domestic real estate markets are still in trouble. In Europe, the economic and employment situation is even worse, creating greater opportunities to find under-valued properties.

And many of the underlying mortgages in CMBS are set to come due in the next three years, meaning many will have to refinance. That will be more difficult if interest rates rise and would create more price volatility and dispersion—just what traders love.

"We love the opportunity—we think there's value in plain sight," said Warren Ashenmil, who recently founded CMBS-focused hedge fund firm Jerica Capital.

Ashenmil, who previously was a portfolio manager at Tricadia Capital, plans to begin trading for his new hedge fund in the first quarter and believes he can generate returns of about 15 percent net of fees for investors.

(Read more: How DC mess could curb commercial real estate)

Another new entrant is Stephen Feinberg's Cerberus.

The firm has traded CMBS since April 2008 and earned double-digit gains every year besides 2008, including 28.7 percent gross in 2013 through October, according to investor materials obtained by CNBC.com. Given those returns and investor demand, Cerberus launched a dedicated vehicle, the Cerberus CMBS Opportunities Fund, on Oct. 7. Led by Scott Stelzer, it had $66.4 million in assets at the end of October.

Cerberus declined to comment through a spokesman.

Other firms that increased their exposure to CMBS have performed well overall.

Ellington roughly doubled its exposure over the year to as much as 10 percent of the strategy currently, according to Huang. The Ellington Credit Opportunities Fund, which trades CMBS among other types of debt, is up 14.46 percent net of fees this year through October, according to a report by HSBC's Alternative Investment Group.

Huang said he targets returns of between 6 percent and 17 percent gross return, depending on the risk of the CMBS tranche. "CMBS offers good relative yield," he said.

Philip Weingord's Seer Capital Management is up 10.48 percent through November with about 26 percent of its portfolio in CMBS as of October. It was 24 percent in January, according to a person familiar with the fund. A spokeswoman for Seer declined to comment.

CMBS has been about 14 percent of the book at Chris Hentemann's $833 million 400 Capital all year. The 400 Capital Credit Opportunities Fund is up 12.64 percent net of fees through October, according to an investor update. A spokesman for 400 did not respond to a request for comment.

Others believe the greatest value is in Europe, where CMBS prices are even lower than in the US.

"European CMBS is attractive both relative to the U.S. and other asset classes in general," said Jason Walker, portfolio manager for the $2.4 billion CQS ABS Fund. "Fundamentals in the sectors are turning. It's an attractive outlook as the underlying real estate environment improves in Europe, especially in the U.K. and Germany."

CQS increased its European CMBS exposure from about 15 percent earlier in the year to roughly 25 percent now. Walker believes the investments can earn between "high single-digit" and "mid-teen" returns. The ABS fund is up 9.32 percent this year through November, according to performance information obtained by CNBC.com.

Build it and they will come?

Despite that excitement, it's not clear if investors will come.

Hedge fund data tracker eVestment projects funds that run CMBS strategies won't raise much money in 2014.

The firm said that the mortgage funds it tracks had net outflows of $12.1 billion over the first three quarters of 2013. And search activity in the eVestment database for MBS strategies has remained neutral over that past 12 months, which is "a significant indication of flat asset-flow activity heading into 2014," a spokesman for the firm said.

(Read more: The new mortgage landscape: What you need to know)

"It seems like a good opportunity without too much risk, but it's not like I'm jumping up and down about it," said one fund of hedge funds manager who invests with many mortgage-focused funds and asked not to be named.

Part of the reason is the potential returns are good—but not great. Manus Clancy, a senior managing director at CMBS information company Trepp, said "B" rated slices of the securities—some of the riskier ones—are now yielding around 15 percent, but that assumes no losses on the underlying real estate.

One to point out the risks of CMBS recently was Paul Singer of Elliott Management, which has some exposure to the market.

"The rise in rates shows no signs yet of dampening commercial real estate prices or transaction activity, but we suspect that will change if interest rates move up materially from current levels," Singer explained in a recent letter to investors.

Regardless, few doubt that there's some opportunity—even if the returns targets aren't electrifying.

"We expect many will continue to look at distressed commercial opportunities, especially those being created by large retailers with vast real estate assets like JC Penney and Sears, to name just a couple," said Brian Shapiro, CEO of Simplify, a New York-based advisory and data firm that tracks hedge funds. "The outlook is moderately positive."

—By CNBC's Lawrence Delevingne. Follow him on Twitter @ldelevingne.