Kerry and counterparts from the five permanent members of the U.N. Security Council and Germany described their meeting with Zarif on the sidelines of the U.N. General Assembly as "a change in tone" from encounters with Iran's previous, hard line government.
European Union foreign policy chief Catherine Ashton, who chaired the meeting, told reporters it had been "a substantial meeting. Good atmosphere. Energetic."
(Watch now: What does Iran think?)
She said the two sides had agreed on an "ambitious timetable" to address Western concerns and would meet again in Geneva on Oct. 15 and 16 "to pursue the agenda to carry on from today's meeting and to hopefully move this process forward."
Ashton also added a note of caution, saying it was important to focus on "effective work that we do on the ground."
The New York talks involved the five permanent members of the U.N. Security Council - Britain, France, Russia, China, the United States - and Germany, known as the P5 1.
Iran's new president is a moderate cleric who has stepped up efforts to improve Iran's image abroad during his visit to New York this week and has said Iran will never develop nuclear weapons.
U.S. President Barack Obama on Tuesday cautiously embraced Rouhani's gestures as the basis for a possible nuclear deal and challenged him to demonstrate his sincerity.
But the failure to orchestrate a handshake between the two leaders, apparently because of Rouhani's concerns about a backlash from hardliners at home and perhaps Obama's concerns about the possibility of a failed overture, underscored how hard it will be to make diplomatic progress.
Even without making any real concessions so far, Rouhani has offered a softer, more reasonable tone than his stridently anti-U.S. predecessor, Mahmoud Ahmadinejad.
The sanctions imposed on Iran have begun to bite severely. Iranian oil exports have fallen by about 60 percent in the past two years as the EU stopped purchases and most Asian buyers drastically cut imports because of the sanctions. Iran is now earning only around $100 million from oil sales a day as opposed to $250 million two years ago.
(Watch now: Can Iran's new president break the nuclear impasse?)
The six powers said in February they wanted Iran to stop enrichment of uranium to 20 percent, ship out some stockpiles and shutter a facility where such enrichment work is done. In return, they offered relief on international sanctions on Iran's petrochemicals and trade in gold and other precious metals.
U.S. officials say that offer remains on the table. But signaling some of the obstacles that could hamper any new diplomacy, Iran on Thursday sharply criticized the U.N. nuclear watchdog over "baseless allegations" about its atomic activity.
It was an apparent reference to the International Atomic Energy Agency's concerns, spelled out in a series of quarterly reports, about what it calls the possible military dimensions to Iranian nuclear activities.
The Fordow uranium enrichment facility Kerry referred to is a secret plant buried 299 feet (91 metres) underground, making it difficult to bomb should the West or Israel decide they need to take military action against Iran's nuclear program.