When U.S. financial markets crashed in 2008, the Federal Reserve began buying billions of dollars worth of agency mortgage-backed securities (loans backed by Fannie Mae, Freddie Mac and Ginnie Mae). As part of the so-called "taper" in 2013, it gradually stopped using new money to buy MBS but continued to reinvest money it made from the bonds it had into more, newer bonds.
"In other words, all the income they receive from all that MBS they bought is going right back into buying more MBS," wrote Matthew Graham, chief operating officer of Mortgage News Daily. "Over the past few cycles, that's been $24-$26 billion a month — a staggering amount that accounts for nearly every newly originated MBS."
At some point, the Fed will have to stop that and let the private market back into mortgage land, but so far that hasn't happened. Mortgage finance reform is basically on the back-burner until we get a new president and a new Congress. As long as the Fed is the mortgage market's sugar daddy, rates won't move much higher.
Read MoreWhat a Fed rate hike means for you
"Also important is the continued popularity of US Treasury investments around the world, which puts downward pressure on Treasury rates, specifically the 10-year bond rate, which is the benchmark for MBS/mortgage pricing," said Guy Cecala, CEO of Inside Mortgage Finance. "Both are much more significant than any small hike in the Fed rate."
Still, consumers are likely going to be freaked out, especially young consumers, if mortgage rates inch up even slightly. That is because apparently they don't understand just how low rates are. Sixty-seven percent of prospective homebuyers surveyed by Berkshire Hathaway HomeServices, a network of real estate brokerages, categorized the level of today's mortgage rates as "average" or "high."
The current rate of 4 percent on the 30-year fixed is less than 1 percentage point higher than its record low. Fun fact, in the early 1980s, the rate was around 18 percent.
"The Fed is seeing more people going back to work, and with the expectation of job growth for America it feels comfortable with its intent to raise rates," said Berkshire Hathaway HomeServices President Stephen Phillips. "But the reality is that an entire generation of first-time buyers has never experienced a meaningful rate increase. This is a new and unfamiliar phenomenon to them."
An increase from the current 4 percent on the 30-year fixed to 4.25 percent on a loans ranging from $200,000 to $300,000 would amount to less than the average borrower probably spends at Starbucks every month. Still, a majority of respondents to the Berkshire survey, which was mostly millennials and Gen Xers, said rising mortgage rates would make them "anxious" about their current financial situations.
Another survey by real estate brokerage Redfin appeared less dire, claiming its buyers said they were "unfazed" by the prospect of rising rates. Just 6 percent, according to the report, seemed to care at all. They were far more concerned with rising home prices and the very tight supply of homes for sale.
The exception, of course, was among what Redfin referred to as "budget-conscious buyers shopping for the least-expensive houses." Redfin categorized them as buyers looking at homes priced $250,000 or less. Seventy-one percent of them said they were concerned.