Chiba Bank, one of the few banks that discloses net interest margins (NIM) on their lending, saw its NIM on mortgages tumble to 1.44 percent from 2.32 percent between the second-quarter of fiscal 2007 and 2014, according to BNP Paribas.
Japan's three biggest banks – Mitsubishi UFJ, Mizuho and Sumitomo Mitsui Financial Groups trimmed rates on 10-year fixed rate mortgage to a fresh low of 1.1 percent on Monday, according to a Nikkei report, as they attempt to lure home buyers.
The race to the bottom in mortgage rates comes amid a steady decline in lending rates; long-term prime lending rates fell from their recent peak above 2.5 percent in 2006 to an all-time low of 1.1 percent in December 2014, Bank of Japan data show.
Read MoreJapan's consumer inflation eases for 5th month
"Banks had pinned their hopes on locking in new mortgage buyers," with the aim of eventually cross selling products such as mutual funds and insurance policies to new borrowers, said Barclays bank analyst Shinichi Tamura.
But that hasn't panned out.
In April 2014, Japan raised its consumption tax to 8 percent from 5 percent. Homebuyers frontloaded their purchases ahead of the tax hike, according to BNP Paribas' Sameshima, noting new mortgages have declined since then. At major mortgage provider Sumitomo Trust & Banking, for instance, the number of new mortgages dropped 27 percent on-year in the third-quarter, according to Sameshima.
Capital gains on stock holdings have fueled profits at Japan's banks after the Bank of Japan's latest round of monetary easing in October pushed the Nikkei up 7.9 percent between October and December 2014. Over the past week, Japan's three biggest banks reported above-view fiscal third quarter profits.
But the latest fall in interest rates could undercut those gains: a 0.05 percentage point drop in mortgage rates will wipe out profits of 60 billion yen ($509.6 million) at the 120 member banks of the Japanese Bankers Association, according to BofA Merrill's Otsuki.
With Japanese government bond yields near record lows, "who knows how much lower mortgage rates will fall or for how long," said BNP Paribas' Sameshima.
Still, there are "some signs that margins are bottoming out," he said. Overall, loan volumes grew by 2.2 percent in the second-quarter, although lending margins didn't. But, in theory, "stronger demand should lift margins," he said.