Weaver said refinance clients who were happy to float rates just three weeks ago are now considerably more afraid of where rates will go. They're willing to take money off the table to lock in now.
The impact is equally deep on potential homebuyers. This is the heart of the busiest season for sales, and now potential buyers, already highly sensitive to rising home prices, have something else to worry about.
Read MoreMetro train transforming this DC suburb
"If the Wednesday surge of Treasury yields persists, the impact on mortgage rates is likely to result in a bout of affordability shock to many housing markets across the country," said Len Kiefer, deputy chief economist at Freddie Mac.
The shock may already be underway. Mortgage applications to refinance, which are the most rate sensitive, plunged last week, even though rates slid slightly. The fact that the 4 percent range is the new normal has borrowers pulling back. Even mortgage applications to purchase a home fell, down 3 percent last week, as potential homebuyers recalculated that ever-important monthly payment.
That calculation is far more rate sensitive than these small weekly moves might indicate. Take the first quarter of this year, for example.The average down payment for single family homes, condos and town homes purchased in the first quarter was 14.8 percent of the purchase price, down from 15.2 percent in the previous quarter and down from 15.5 percent a year ago to the lowest level since the first quarter of 2012, according to a new report from RealtyTrac.
Read MoreWeekly mortgage applications plunge 7.6% on higher rates
"When rates plunge, end-user house demand increases quickly—ergo the Q1 2015 outperformance year over year—and then when they surge, the opposite happens—ergo a year earlier when demand was dead," said Mark Hanson, a California-based mortgage analyst.
The average rate on the popular 30-year fixed mortgage is now up three-eighths of a percentage point since mid-May to about 4.125 percent. While three-eighths may not sound like a lot, it's not the actual number, or the $40 or $50 more on the monthly payment, but the new trend higher.
"It's more of what it's going to look like, where we're going. Is it a train that's not stopping? When we see an eighth, a quarter, now it's starting to become typical language," said Weaver. "We would normally see a little bit of a pullback, and then it goes up again, and now that's not happening. We're slowly steadily increasing."