But many officials who enjoyed working with Mr. Summers remain in prominent positions within the White House. Those include Gene Sperling, the current head of the National Economic Council, the position Mr. Summers occupied for the first years of the Obama administration; Brian Deese, deputy director of the budget office; and Jason Furman, who won Senate confirmation on Thursday to become chairman of the Council of Economic Advisers.
Others agreed that Mr. Summers could at times be brusque, but that he also welcomed being challenged.
"His favorite line is 'Tell me why I'm wrong,' " said Stuart E. Eizenstat, a former deputy Treasury secretary. Ms. Sandberg echoed that anecdote, recalling how Mr. Summers would ask his colleagues at Treasury to come up with the strongest argument against their own ideas.
People close to White House officials said that many of them considered Mr. Summers perhaps the most brilliant economic policy mind of his generation. One former member of the White House economic team has even taken to using a certain shorthand to denote someone of truly superior intellect: "Larry-smart."
(Read More: The wheels keep turning: Pros handicap Fed pick)
His deep understanding of the economy, concern with unemployment and ability to manage complexity and crisis would make him a stellar Fed chairman, they argue, also noting how much Mr. Obama likes how Mr. Summers's mind works.
"There's going to be an international crisis at some point, and he's the guy you want in the room," said an economic policy expert who declined to comment on the record.
Mr. Summers set himself apart as an economic thinker among Mr. Obama's advisers and staff members during the 2008 campaign, former colleagues said.
"Larry had a very good way of synthesizing information, boiling it down to what it all meant and what the consequences were for the economy," said Stephanie Cutter, who worked on both Obama campaigns. "Larry was the only one who had been through a financial crisis before," she added. "He was speaking from a place of authority."
White House insiders said that Mr. Summers's firsthand management of the Asian financial crises, and the influence of his ideas managing the downturn in 2009, made them more confident at his ability to handle problems that might confront the American economy in the next few years.
"If I were confident that we were going to get a smooth recovery that is faster than it has been in the past few years, Janet, with her ability to manage consensus, would be a better choice," said J. Bradford DeLong, a professor at the University of California, Berkeley, who worked with Mr. Summers in the Treasury Department in the 1990s and has written academic papers with him.
"But it's a strange and complicated world, in which creativity and the ability to think outside of the box might be more necessary and a Fed headed by Larry might be more successful than one headed by Janet," Mr. DeLong said.
Among academics, Mr. Summers was already considered one of the finest economists of his generation before joining public life. He is an "economist's economist," said Lawrence Katz, a Harvard professor who has written several papers with Mr. Summers.
"There wasn't one idea or one subject where he spent 20 years thinking about it to write one great paper or one great book," he added. "He has 55 really good ideas, on monetary economics, on tax, on savings, on investment, on unemployment. Those papers stood the test of time; we still teach them even though he hasn't written an academic paper for 20 years."
—By Annie Lowrey of The New York Times. Binyamin Appelbaum contributed reporting.