Investors are moving out of housing. Here's why

They swarmed the distressed housing market, buying thousands of foreclosed properties and pushing prices higher faster than anyone expected. Now investors are pulling back, dissuaded by the higher prices they themselves brought about.

"Perhaps the numbers aren't working out," said Lawrence Yun, chief economist of the National Association of Realtors, which reported that just 15 percent of June sales were by investors. That is the lowest share since the association began tracking this cohort in October 2008.

Current homeowners are now driving the housing market, as even investor traffic fell in June for the fourth straight month, according to Campbell/Inside Mortgage Finance. That could mean slower sales going forward, as still tight inventory keeps move-up buyers in place. That, and negative home equity.

GP: House with Price Reduced sign real estate
Daniel Acker | Bloomberg | Getty Images

David and Heather Littlejohn are one of about 10 million underwater borrowers in the U.S. Millions more don't have enough equity to afford a move up. The Littlejohns would love to sell the Oregon home they built as newlyweds in 2005 and move to a larger one, but that would mean paying into their mortgage.

(Read more: House-flipping flourishing again)

"It is really frustrating. We've had several friends that have gotten into larger homes, and had we known we would still be here several years later, we probably would have designed this home differently," said Heather.

At the height of the foreclosure crisis, investors—some individual and some larger funds—were making up more than a third of home buyers. Most set their sights on the West, where the crisis hit hardest. That is why prices in that region are up more than 20 percent now from a year ago, but are still way off from where they were at the peak of the boom.

"Everybody else seems to be getting out OK on this one, and here we are just the perfect timing and circumstance to be on the outside looking in," said David. "There's this theoretical wealth creation all around us, and yet we're not participating in it, so, yeah, it's pretty frustrating."

First-time homebuyers could say the same. Usually about 40 to 45 percent of the market, they made up just 29 percent of buyers in June, according to the National Association of Realtors. A lack of supply has made the market far too competitive for these buyers, who usually need financing and have smaller down payments.

(Read more: Map: Tracking the US real estate recovery)

"We can thank investors for that limited inventory, of course, as many entry-level buyers are now going to have no choice but to rent," said Peter Boockvar of The Lindsey Group.

Even as investors move out, cash is still king in this market. Thirty-one percent of sales were all-cash. That share is usually below 10 percent. June's home sales were largely unaffected by the recent rise in mortgage rates, as contracts for those sales were signed in April and May.

The Realtors' group expects higher rates to slow sales in the coming months, and if investors, who drove the market for so long, continue to exit, those sales could be even slower.

(Read more: Go green? Get a bigger mortgage)

By CNBC's Diana Olick. Follow her on Twitter @Diana_Olick.

Questions? Comments?