Over the four weeks to March 25, Japanese investors bought a record 5.875 trillion yen worth of foreign bonds, while selling 1.827 trillion yen worth of Japanese bonds in the week to March 25 alone, according to data from CIBC.
"We don't know who bought these foreign bonds, but obviously, the most likely candidates are insurance and pension funds," Thieliant said. "The negative rates have pushed them out of Japan."
So far, selling of government bonds varies across different types of institutions. Investment trusts and insurance companies have been net sellers of government bonds for at least 20 months through February, the latest month with available data, according to the Japan Securities Dealers Association.
There are already signs that some investors are looking to diversify away from Japan's government bond market.
A Reuters survey of Japanese fund managers, published Thursday, found they reduced their Japanese debt holdings in their model portfolios to 39.9 percent in March from 45.7 percent in February.
It's hard to find an excuse to buy the bonds, noted Takuji Okubo, chief economist at Japan Macro Advisors.
"It's an asset with volatility risk with no return, or even a negative return," he said, although he noted some "flow traders" might want to buy and hold until they have a chance to sell to the BOJ.
Okubo also noted that expectations the BOJ would cut interest rates further into negative territory are fading. That suggests the yields won't keep falling.
"People are starting to feel the BOJ realized that lowering interest rates further doesn't really help the economy and there's significant international opposition to such a move," Okubo said. "There's a mindset in the market that the BOJ is done with lowering the policy rate further. If the policy rate is unlikely to go deeper into negative (territory), there's no reason to hold JGBs."
The BOJ's foray into negative rates doesn't appear to have had the desired impact on markets, with the yen strengthening, rather than weakening. Additionally, on Friday, the BOJ's Tankan survey of big manufacturers showed confidence worsened more than expected in the first quarter, with the decline in part due to the negative-rate policy.
But while some banks are continuing to buy, institutions' interest in owning JGBs may wane.
"The only reason they would hold JGBs is you want to ensure you have Japanese yen in 10 years," Steve Goldman, managing director at fixed income manager Kapstream Capital, said last week.
—Nyshka Chandran contributed to this article
Follow CNBC International on Twitter and Facebook.
—By CNBC.Com's Leslie Shaffer; Follow her on Twitter @LeslieShaffer1