There are no clear statistics on the number of foreign household helpers in Japan as many are working informally and those working legally, do so under a broad visa category. But foreign workers themselves say their numbers are shrinking.
"It has gotten much harder since I first came in 1990 on a tourist visa to look for work," said a 69-year-old housekeeper from the Philippines. She has a work visa - but on a passport bearing her dead sister's name.
She said she was forced to leave Japan a few years ago because authorities learnt she was no longer employed by her previous visa sponsor. So she said she was forced to resort to using her late sister's unblemished paperwork to get back into Japan.
Her employer, an American executive, had hoped to hire a Japanese housekeeper.
(Watch this: Is 'Abenomics' good for Japan's economy?)
"I couldn't find anyone who would commit to full-time work and was willing to perform multiple job duties, from childcare to cleaning to marketing," she said.
Abe's plan to get more women working focuses on expanding the number of daycare centers. But Japanese women are finding that daycare centers do not stay open to match the long hours they need to adopt to compete in a male-dominated workplace.
Japanese domestic-help services exist, but many limit the hours and duties of their workers.
Foreign helpers tend to be willing to work for less and are more flexible, but only foreign diplomats and expatriates with an elite visa status can offer legal visa sponsorship and employment.
"The fact that I, as an American national and a foreigner, can sponsor a foreign domestic helper, yet my Japanese peers cannot, is just mind-boggling," said Kathy Matsui, chief Japan strategist at Goldman Sachs.
(Read more: Verdict on Abenomics, one year on)
She estimates that raising women's participation in the labor force to 80 percent, matching men, could lift Japan's gross domestic product by as much as 14 percent.
"The demand is clearly there, the supply exists, but given all of the strict immigration rules here, Japan is not the obvious destination for many of these domestic helpers," Matsui said. "It's as if the government is preventing these supply and demand curves from meeting."
Underlining the point, Japanese and foreign domestic workers comprise less than 0.1 percent of the labor force, the International Labor Organization, a United Nations agency, estimates. That compares with about 0.5 percent in the United States and 7.7 percent in Hong Kong.
The American Chamber of Commerce in Japan urged the government in June to revise its immigration laws to let citizens and permanent residents with household incomes of 7 million yen ($68,200) or more to sponsor household help.
(Read more: Japan retail sales, household spending beat forecasts)
"If you keep the doors open, there are going to be, legitimately, Japanese young families who will be able to employ foreign domestic workers," said Kumi Sato, president and chief executive of public relations firm Cosmo in Tokyo, an author of the U.S. business proposal.
She said some Japanese families already hire foreign workers illegally, so the change would help legitimize some existing arrangements that fall into grey areas.