The trade tariffs hurt companies that buy steel — including automakers, which employ about 6.5 million Americans — to protect steelmakers, which employ about 140,000, Summers added.
Earlier on CNBC, current Commerce Secretary Wilbur Ross defended the tariffs, saying the stock market will "adjust" to any changes in U.S. trade policies.
Summers responded: "To defend a wanton act of economic aggression that hurts our economy by saying that all things work out in time is absurd. I'm appalled by the action."
The announced import tariffs of 25 percent on steel and 10 percent on aluminum, which go into effect at midnight Thursday, hurt America's relationship with key allies, he said.
Ross spoke about the tariffs' impact on CNBC. "In terms of an automobile, it's also a fraction of 1 percent. And for the economy overall, it's a very small fraction of 1 percent."
"First of all 1 percent of GDP is $160 billion" and "that's a huge amount," Summers countered, arguing Ross' claim that beer, soft drink and soup cans would only go up less a penny misses the point. "Raising the price of steel by 25 percent adds a lot more than fractions of a penny to the price of domestically produced automobiles."
The trade action especially puts the U.S. talks with Canada and Mexico on an overhauled NAFTA in a tough spot, Summers said.
Ross also told CNBC earlier that ending the tariff relief on Canada and Mexico reflects the lack of progress in those talks to update the 1994 North American Free Trade Agreement.
The president has repeatedly said he would pull the U.S. out of NAFTA if a better deal can't be worked out.
The steel and aluminum tariffs were originally announced by Trump on March 1. The EU, Canada and Mexico were given temporary waivers, set to expire Friday. They are expected to retaliate.
Summers did acknowledge that more dire predictions of a recession due to the president's trade polices are off base. The economy is currently strong, he added, but contended it has nothing to do with Trump. It's more of a legacy of historically low interest rates that were put in place after the 2008 financial crisis aimed at boosting growth, he said.
— The Associated Press contributed to this report.