For the homebuyer, take for example a $200,000 loan. At the lows of this year—around 3.6 percent versus the current 4.25 percent—the difference in the average monthly payment is $72 (excluding taxes and insurance), according to Mortgage News Daily's Matthew Graham. That may not sound like a lot, but it has plenty of ramifications beyond just the extra dollars.
Read MoreMortgage rates: 'Definitely in panic mode'
"It can create an unforeseen issue with qualifying for the loan in the first place," Graham said. "Many borrowers are just squeaking in under the debt-to-income ratio guidelines. So a modest increase in the monthly payment can cause the ratio to rise just enough to kill the deal. If those borrowers have the means, they can 'buy down' the rate with upfront points, but we're talking about thousands of dollars there. If you had to buy down all the way to 3.625 from 4.25, that would cost around $16,000 on a $200,000 loan. No one really does that, but many have to buy down to, say, 4 percent, which costs about $3,000 in the same example."
For sellers, the rise in mortgage rates means their customers have less spending power. The majority of the nation's large housing markets are experiencing tight inventory of homes for sale, and that has put sellers decidedly in the driver's seat. If buyers, however, can't compete anymore, then sellers lose their leverage. Homebuilders might be better positioned, as most of the large public ones have their own lending arms and can work closely with buyers to make the math work. Regular sellers, though, may have to reconsider their asking prices in a higher interest rate environment.
Higher rates therefore could also affect the direction of home prices.
Read MoreHow tiny mortgage rate moves can buy you a lot
A recent analysis by John Burns Real Estate Consulting looked at the typical American family, earning $60,000 a year. They can afford about $1,800 a month for the mortgage payment, given a normal amount of other debt. For a 30-year fixed-rate loan, back in 2000, when rates were 8 percent, that would have qualified them for a $245,000 loan. At 4 percent, which is where rates are today, they can qualify for a $377,000 loan.
"Each 1 percent drop in interest rates in the last 15 years has allowed home sellers to raise price 12 percent," according to the firm.
Now take that theory to the flip side. Higher rates mean home prices could fall just as easily. Whatever the supply issues, affordability trumps everything in housing.