"[McMillon] is not fit to be the CEO of Wal-Mart if his judgment is so bad as to suppose that being a member of this council is giving him some kind of effectiveness," Summers argued. "What he may be — and what a number of CEOs are — is scared; scared that if they leave, the president using the tools of government will retaliate."
If Trump has created a culture of fear among American CEOs, "that is a particularly damning indictment far beyond any in Charlottesville of his presidency," Summers said.
Wal-Mart declined to comment on Summers' remarks.
On Monday, Merck CEO Ken Frazier resigned from the White House manufacturing council, igniting the debate about whether corporate leaders should leave Trump's advisory panels over the president's response to Charlottesville.
Less than 24 hours later, Intel CEO Brian Krzanich and Under Armour CEO Kevin Plank followed suit.
On Tuesday, AFL-CIO President Richard Trumka, union colleague Thea Lee, and the president of the Alliance for American Manufacturing, Scott Paul, resigned as well.
"I cannot understand why others have not followed Ken Frazier out the door. I cannot understand why they have not endorsed his courageous action," said Summers. "This is not a happy day for American business."
Summers spoke before a member of Trump's Strategic and Policy Forum told CNBC the panel has agreed to disband. Also Wednesday, Campbell Soup chief Denise Morrison and 3M CEO Inge Thulin resigned from Trump's manufacturing council.
In addition to calling out corporate America, Summers also said Wednesday he's surprised no officials of Trump's administration had resigned.
"There's a tradition in America of people of principle when they are sufficiently offended and disagree with their president on matters fundamental resign," he said.
"I have been disappointed that there have been no resignations on principle by the political appointees of this administration."