European investors began buying JGBS during the past few weeks, in anticipation of the ECB move, and the trend became more pronounced once the central bank's bond buying began, with yields turning negative on many European government short-term notes.
For now, investors appear to be concentrating on buying short-term Japanese notes, but their interest could extend to longer-dated debt as euro zone yields such as those on 10-year notes also turn negative, market players said.
While the yield on German five-year debt has turned into negative territory, and the French five-year yield has recovered to barely above it, five-year JGBs offered a positive yield of 0.125 percent as of Tuesday.
Read MoreJapan pines for Australia's yields
"Foreign investors' demand is coming back, Particularly in the two to five-year maturities, where German bund yields are deep into negative, JGBs look relatively attractive," said Soichi Takeyama, strategist at SMBC Nikko Securities.
In the week through March 7, foreign investors bought a net 673 billion yen of Japanese bonds and bills, according to Ministry of Finance data.
In addition, European investors could also earn extra basis points if they swap yen proceeds to the euro. For instance, they could have gotten around 39 basis points to 5-year Bunds, according to market participants.
"With the ECB beginning its quantitative easing, there's definitely a sense that funds are flowing out," said Mitul Kotecha, head of FX strategy, Asia-Pacific for Barclays in Singapore.
"It's not a one-way flow, but there is this expectation with ECB QE that there is going to be substantial outflows from Europe, as QE intensifies," he said.
To be sure, JGB yields aren't as appealing as those on U.S. Treasuries. The yield on the U.S. 5-year note is over 1.5 percent, and is poised to go higher on market expectations that the U.S. Federal Reserve could hike interest rates as early as June.
Still, Japanese market participants have to get used to the idea that their perennially low-yielding debt is a potential play for investors seeking higher yields.
"I never thought this day would come," said a fixed income fund manager at a European asset management firm in Tokyo.