When stocks sell off, investors historically head to the bond market because it is considered a safer investment. Higher demand means lower yields. Lenders price according to the yields on mortgage-backed bonds, which generally follow the 10-year Treasury.
Mortgage rates do not follow the Federal Reserve funds rate, but most expected that as the Fed raised rates, mortgage rates would rise as well. This has more to do with an improving economy, which would be behind both.
"Given the Fed rate hike and strong ADP data yesterday — among other reasonably decent economic anecdotes — we would be more justified in expecting bonds to be under pressure at the start of the year," added Graham, calling the drop in rates, "a pleasant surprise."
Read MoreMortgage applications plunge after earlier rate-hike rush
The average rate on the popular 30-year fixed mortgage is now just below 4 percent for the most credit-worthy borrowers. Applications to refinance a loan had dropped dramatically in the last two weeks of 2015 amid higher interest rates, but this move lower could create a new opportunity for thousands of borrowers who have yet to refinance at a lower rate.
There are not many regular borrowers who would benefit from the current rates, given the refinance boom of the last three years, when rates were hovering around record lows. There are, however, nearly 430,000 borrowers who could still benefit from the government's HARP refinance program, according to the Federal Housing Finance Agency. This is for borrowers who still owe more on their mortgages than their homes are worth, commonly known as "underwater." Their loans must be government-backed. Why so many still?
"They may be in a good financial position, able to make their monthly payments and don't want to mess with it," said Andrew Wilson, Fannie Mae's chief spokesman. "There are always some number of people that just never do, and the question is why not?"