A potential stall in home price gains and a large drop in the number of distressed properties have some big investors pulling out of the single-family rental market.
They are getting out at the same time that billions of investor dollars continue to pour in.
"I think the investor market is largely past us," Doug Lebda, chief executive of Lending Tree told CNBC. "People were buying investment properties three, four, five years ago. What I hear is that's slowing now."
Recent reports that Oaktree Capital Group is selling about 500 of its homes added fuel to other reports that Och-Ziff Capital management is selling its homes as well. Both declined to comment on the reports. Carrington Mortgage Services stopped buying distressed homes late last year, claiming the market was "a bit too frothy."
(Read more: Investors cooling on 'REO-to-rent')
Home prices are up over 12 percent from a year ago, according to CoreLogic, but still down 18 percent from their peak in 2006. Investors certainly played a role in putting a floor on home prices and then pushing them higher than many predicted.
Now, faced with higher mortgage rates and weak wage and employment growth, even usually bullish brokers predict home prices will stay flat through 2014.
Critics say without rising prices, the rental trade is a low-to-mid single-digit return proposition. Management of the homes can be as tricky as it is costly, and that alone lowers profit dramatically.
(Read more: Apartment demand keeps climbing)
"Investors who were buying REO [bank-owned homes] four and five years ago have the added cushion of home price appreciation to augment returns. But if you've been buying REO or even new homes for rent in the past year or so, the embedded home price appreciation is limited," said a mortgage industry insider who did not want to be identified. "It is going to be very hard for investors to make money on rental fees alone. Looking at the dismal data for household formation, jobs and consumer income, it seems pretty obvious that 2013 may be the peak."
Institutional investors have poured a collective $20 billion into as many as 200,000 properties, or up to 12 percent of distressed home sales over the past 18 months, according to a report from KBW.
It is a tiny slice of the housing stock, to be sure, but investors who remain in the game say it will get larger and the potential for long-term profit is big.
"We don't see it as a trade; we see it as a business," said Justin Chang of California-based Colony Capital. Colony owns over 15,000 homes and is buying at a rate of about 1,000 homes per month. "There is plenty to buy," added Chang.
The number of homes in the foreclosure process was down 34 percent in August year-over-year, according to Lender Processing Services. That still represents more than 1 million homes, while more than 2 million homeowners are behind on their mortgages.
"We're looking at the multiple listing services, we're still looking at REO from the banks, we're looking at short sales, we're even buying some traditional houses now where people are just putting them on the market," said Laurie Hawkes, president and COO of Arizona-based American Residential Properties, a publicly traded real estate investment trust. "We think that if you get a reasonable cost of capital, both debt and equity, you can actually not only create a very attractive return on a current basis, but in today's market, the house price appreciation that we think is still in the market is extraordinary."
American Residential has already bought 80 portfolios of rental homes from smaller aggregators, according to Hawkes, who said that while Arizona is "fully invested," the company is now setting its sights on Georgia, the Carolinas, Indianapolis and Chicago.
"I think soon there will be consolidation from potentially other players who might have had private equity, who can't make it work, whether it's the Carringtons or others in the universe who have decided this isn't for them because they aren't going to make the commitment to the operations or they couldn't make it work," said Hawkes.
(Read more: Shutdown freezes housing's 'pipeline')
REITs may be better positioned in the rental market because they can deliver ongoing dividends through rental revenue, while private companies may be looking to see their biggest returns by reselling the homes. It appears the divide is between the short- and the long-term view.
"We are building a business for the long term and not planning bulk sales," said a representative of Blackstone, the largest institutional investor in the group. Blackstone owns about 32,000 homes in 12 states, according to its filings.
Most investors have depended on private capital until now, but with banks warming to the idea and several of the companies going public, some say there is real room for growth.
"We believe the sector has the potential to emerge as a long-term institutional asset class," said KBW researchers.
(Read more: Map: Tracking the recovery)
Rental demand is strong, with vacancies near record lows and rent growth continuing to gain. Younger Americans are not moving into home ownership as fast as previous generations, and the supply of distressed properties is still elevated.
"The cohort of potential buyers at the younger age of the spectrum really isn't materializing, because so many are locked into student loan debt," said Rick Sharga of Auction.com. "I do think we are going to continue to see a healthy rental market."
Single family rentals have always been a large component of the housing market. The only difference now is that mom and pop landlords are being replaced by large-scale investors and management companies.
"What we've sought to do is to say, 'Could we convert this business into something that looks, acts and smells like multifamily," said Hawkes.
The answer appears to be, yes.
—By CNBC's Diana Olick. Follow her on Twitter @Diana_Olick.
Questions?Comments? facebook.com/DianaOlickCNBC.