Police said the men, aged 24 and 25, were arrested after a raid on a home in a western Sydney suburb on Tuesday and had been charged with planning a terrorist act.
"When we did the search of the premises, a number of items were located, including a machete, a hunting knife, a home-made flag representing the proscribed terrorist organisation IS, and also a video which depicted a man talking about carrying out an attack," New South Wales Deputy Police Commissioner Catherine Burn told reporters.
Read MoreAustralians take to Twitter to support Muslims
"We will allege that both of these men were preparing to do this act yesterday," she said.
The men were not known to police, Burn said.
"This is indicative of the threat that we now have to live with and which we are now having to deal with," she said.
Australia believes at least 70 of its citizens are fighting with the Islamic State group in Syria and Iraq, backed by about 100 Australia-based "facilitators".
The government of Prime Minister Tony Abbott last year committed Australian aircraft and special forces to assist in the battle against IS in Iraq, introduced tough new laws on foreign fighters and gave security forces enhanced powers at home.
"Regrettably there are those out there, some living in our midst, who would do us harm but your government, at every level, will do whatever we humanly can to keep you safe," Abbott told reporters in rural New South Wales.
In December, two hostages were killed when policed stormed a central Sydney cafe to end a 17-hour siege. The gunman, Man Haron Monis, a self-styled sheik who harboured deep grievances against the Australian government and sought to align himself with the Islamic State group, was also killed.