If Iran restores its nuclear program, a military response may be the only option, he told CNBC on Wednesday.
The deal, signed in 2015, was President Barack Obama's signature foreign policy achievement. Other parties to the agreement included Britain, France, Germany, Russia and China.
In exchange for Tehran's promise to limit its nuclear program, the signatories lifted trade sanctions against Iran's economy that had cut its oil exports in half.
Trump made his announcement on Tuesday that the U.S. will withdraw from the deal and restore sanctions on Iran. The new sanctions could start as early as next week, White House press secretary Sarah Huckabee Sanders said Wednesday.
Panetta, who headed the Pentagon and CIA during the Obama administration and is co-founder of the Panetta Institute of Public Policy, said Iran now has no incentive to negotiate.
"We are left with a negotiation that isn't likely to happen," he said. "Or, some kind of potential military confrontation in the future. And that's a lousy choice."
And if Iran decides to move ahead with its nuclear program, Panetta said, "I don't think there is any other option but the potential of some kind of military action."
In addition, leaving the deal will likely fracture relationships with allies, he said.
"We've raised are real questions about whether or not you can trust our country," Panetta said.
"There are a number of crises around the world," he pointed out. "The reality is, the United States cannot deal with these crises without our allies. We can't do this alone. And here we are, taking our greatest allies, France, Great Britain and Germany, and basically throwing them out the window," he said.
The withdrawal will also likely heighten Chinese and Russian distrust of the U.S., he said.
"You can't work out any negotiations without mutual trust," said Panetta.
Panetta said his greatest concern with the Trump administration is that he doesn't "see a strategy here."