Some areas received double the usual September rainfall within 48 hours after tropical storm Etau swept across the central part of Japan's main island of Honshu.
A 63-year old woman was missing after a landslide hit her home, while a man in his 70s in the town of Joso, about 56 km (35 miles) north of Tokyo, was feared trapped in his home when floods swept it away, NHK national television said.
Japan has put heavy emphasis on disaster prevention since the 2011 earthquake and tsunami disaster that left nearly 20,000 dead. It is eager to avoid the criticism faced by previous governments for what was seen as a sluggish response.
"The government will work as one to prioritize the safety of the people and do our best to prevent any further disaster," Prime Minister Shinzo Abe told reporters, and warned of unprecedented rains.
Read MoreJapan machinery orders slide, fueling concerns over economic recovery
Television footage showed people in Joso waving towels while waiting for help on the upper floors of homes engulfed in floods as the Kinugawa river swept through.
Part of a hotel in the town of Nikko, famed for its shrines and temples, had collapsed, the Kyodo news agency said, but there were no reports of injuries.
Rainfall reached 600 mm (24 inches) in the area around Joso, with weather officials expecting at least 200 mm (8 inches) more in parts of eastern Japan, including Fukushima, the site of the nuclear reactor crippled in 2011, before the downpour stops on Friday.