Higher rates have plagued the mortgage market since last summer, when the average rate on the 30-year fixed went from 3.5 percent to 4.5 percent. After a refinance boom throughout much of 2012 and 2013, that market dried up.
Applications to refinance a mortgage were down 70 percent last week from a year ago, according to the Mortgage Bankers Association. And applications to purchase a home were down 14 percent. Total mortgage originations in February hit their lowest level in 14 years, according to Black Knight Financial Services.
"The problem is, even though rates are now falling again, so many people have already re-fied in the last 2½ years. If someone is sitting at even 4.5 percent, they're not going to refi even if rates go to 4 percent. If you have a $200,000 mortgage, and rates only move a half percentage point, it's not in your interest to refinance," said Paul Muolo of Inside Mortgage Finance.
Read More Real estate site targets underserved commercial sweet spot
The question now is, will rates continue to fall, or is this a momentary blip?
"They're still in the same fairly narrow 4.25-4.5 percent range that has characterized most of 2014," said Graham. "We'd need to see another day or two like (Thursday) to be able to say something different is happening."
Most had expected rates to surge higher by now, due to the so-called taper by the Federal Reserve, but so far they have not. As part of the taper, the Fed has been reducing the amount of agency mortgage-backed securities it buys.
"Safe-haven buying has outpaced the Fed's QE3 taper, which is why mortgage rates have stayed low. We saw a similar pattern after the end of the first two QE programs, too. Mortgage rates should remain low so long as inflation rates stay low," said Dan Green, publisher of TheMortgageReports.com.
Green added that mortgage rates remain subject to event risks, be they economic or geopolitical. "Rates could cross 5 percent easily between now and June, or fall into the 3s."
Read MoreDivided housing recovery is all for the 'haves'
While the big banks see their mortgage business fall off, some are shifting strategies.
"They purposely are shedding market share. They seem more than willing to cede business to nonbanks like Quicken," said Muolo, who points out that the banks can make more money on servicing rights, especially on today's better-quality loans. "They're not losing a whole lot of sleep over losing market share in the mortgage business."
Big banks are originating more jumbo loans (loans with balances higher than $417,000), which, given today's tight underwriting, are a very safe bet. Of the 20,000 jumbo loans originated and then securitized since 2010, just two are 60 days past due, according to Fitch Ratings.