A local business owner told NBC News he heard what he thought at first were fireworks.
"But then we took a peek and there were cops firing and the guy went down in front of the building," said the owner, who declined to give his name.
Meanwhile, President Obama reassured a nervous nation that law enforcement was on the case.
"Moments like this, I think it's important to remember what terrorists and violent extremists are trying to do," said Obama. "They are trying to hurt innocent people and create fear in all of us, to disrupt the way we live."
Obama went on to praise New Yorkers and New Jersey residents after their states were once again targeted by terror.
"Folks around here, they don't get scared," he said. "They're tough. they're resilient. They go about their business every single day."
Rahami, who lives in Elizabeth, New Jersey, was identified as a suspect after a fingerprint was found on one of the devices that failed to detonate, a senior law enforcement official told NBC News.
They also found more information that pointed to Rahami on cell phones that were wired to the unexploded bombs, the official said.
"He certainly seemed to do virtually nothing to cover his tracks," the official says.
But the official downplayed any talk of Rahimi being part of a "cell" and said at this point have no idea whether anyone else was involved.
Asked whether the bombings were ISIS-inspired or directed, the official said authorities have no idea: "We're a long way from that."
Still, the FBI warned that Rahami should be considered armed and dangerous. And New York Gov. Andrew Cuomo said officials could not rule out international terrorism.
"Today's information suggests it may be foreign related, but we'll see where it goes," Cuomo said Monday on MSNBC's "Morning Joe."
Rahami is the man seen in surveillance footage taken Saturday night in Manhattan's Chelsea neighborhood, the site of an explosion that injured 29 people, a senior law enforcement official with direct knowledge of the investigation told NBC News.
The source said there is other physical evidence linking Rahimi to the devices that went off or were found in New York and New Jersey.
The development came hours after a backpack that appeared to contain pipe bombs exploded as a police robot examined it near a New Jersey train station. That blast happened shortly before 1 a.m. ET Monday. It was the second in New Jersey since Saturday morning.
Elizabeth Mayor Chris Bollwage told reporters the bomb squad robot was "cutting into the device when it exploded" in his city. A spokesman for Bollwage had earlier described the blast as a controlled detonation.
Officials familiar with the investigation confirmed to NBC News that heavily armed FBI agents spotted at a fried chicken restaurant in Elizabeth around 6 a.m. were involved in the tri-state bombs probe.
Several law enforcement officials told NBC News that they are concerned that an active terrorism cell with multiple players could be at work in the New York-New Jersey area.
Earlier, authorities stopped a "vehicle of interest" in the New York blast at about 8:45 p.m. ET Sunday near the Verrazano-Narrows Bridge in New York, the FBI said. Five people were being questioned by the FBI early Monday, they said.
None of the five people had been charged, authorities told NBC News.
Multiple senior law enforcement officials told NBC News that the suspicious device discovered Sunday night in Elizabeth looked similar in appearance to the one that exploded Saturday morning in Seaside Park, N.J.
The latest package was a backpack, and it was found by two men in a garbage can about 300 feet from the front door of a crowded pub in Elizabeth, Bollwage said.
When they saw wires and pipes, they dropped it and immediately went to police headquarters, he said.
"We do not believe those two are involved," the mayor said. "We believe they did the right thing."
Bollwage said he was "extremely concerned for the residents of the community" if "someone could just go and drop a backpack into a garbage can that has multiple explosives in it."
New Jersey Transit suspended services between Newark Airport and the Elizabeth station, and Amtrak suspended service along parts of the Northeast Corridor.
The explosion Saturday night injured 29 people in Manhattan's Chelsea neighborhood. Less than three hours later, a "possible secondary device" was found a few blocks away on 27th Street while officers were combing the area.