Still, rising rates could not come at a worse time for the housing recovery. Home prices rose over ten percent in March, according to the latest surveys from S&P/Case-Shiller. Every one percentage point rise in mortgage rates reduces the average home buyer's maximum purchase price by 11 percent, figures Green.
(Read More: Home Price Gains Go to Double Digits)
First-time home buyers will be hit hardest by rising rates, just as they were beginning to trickle back into the market. They made up just 29 percent of buyers in April, according to the National Association of Realtors, the lowest level in two years. Historically, they usually account for about 40 percent of the market.
The 30-year fixed mortgage hit a record low rate of 3.47 percent in December of last year. Even though it is still well below historical norms, this small rise is already taking its toll.
"In my world it's clearly slowing the market and pricing. Right now I have properties that are well-priced yet sitting on the market unsold," said David Fogg, a real estate agent in Burbank, CA. "Should rates continue to rise, values will likely soften."
(Read More: Borrowers Becoming Accidental Landlords)
—By CNBC's Diana Olick; Follow her on Twitter
@Diana_Olick or on Facebook at facebook.com/DianaOlickCNBC
Questions? Comments? RealtyCheck@cnbc.com