After a sizable jump in application volume two weeks ago, last week was payback time in the mortgage market.

Total applications fell 4.3 percent week-to-week on a seasonally adjusted basis for the week ending Nov. 21, according to the Mortgage Bankers Association (MBA). This even as mortgage rates dropped.