The company's board of directors elected President and Chief Operating Officer Tim Sloan to succeed him as CEO, while Lead Director Stephen Sanger will serve as the board's non-executive chairman.
The news comes after it was revealed that employees in Wells Fargo's community banking division opened about 2 million accounts without customer authorization, which resulted in the bank paying $185 million in penalties. Stumpf was grilled on Capitol Hill as he defended the bank's sales practices.
Sloan told CNBC late Wednesday that Stumpf "concluded this was the right thing to do for the company to allow it to move forward," but added the work is far from finished.
"There's a lot to be done. No one individual is necessarily going to make the difference," he said. "We all have to work together to restore the reputation of the company to continue to focus on moving the company forward."
Stumpf said in a statement that he was "grateful for the opportunity to have led Wells Fargo" and is optimistic about the bank's future.
"While I have been deeply committed and focused on managing the Company through this period, I have decided it is best for the Company that I step aside," he said in a statement.
A spokesman for Wells Fargo said that there will be "no severance payment or agreement related" to Stumpf's departure.
"There are some retirement benefits that are detailed in our proxy statement. They are not accessible for the next 6 months. That is a normal lag time," the spokesman said in a statement.
Sanger said in a statement that while the retiring CEO helped "create one of the strongest and most well-known financial services companies in the world," Stumpf "believes new leadership at this time is appropriate to guide Wells Fargo through its current challenges."
The board also said that Sloan was elected to the board and that independent director Elizabeth Duke will serve as its vice chair.
Rep. Jeb Hensarling, R-Texas, chairman of the House Financial Services Committee, told CNBC that the committee will continue its investigation "because Wall Street and Washington must be held accountable."
Stumpf's decision to retire was "more than appropriate," said Rep. Maxine Waters, D-Calif, who is the party's ranking member of the Financial Services Committee. She said in a statement that Wells Fargo's actions were "unconscionable" and that Stumpf should bear direct responsibility. Waters remained concerned that Wells Fargo is "too big to manage" and renewed calls to break it up.
Sen. Sherrod Brown, D-Ohio, a member of the Senate Committee on Banking, Housing, and Urban Affairs said in a statement that Stumpf's retirement "does nothing to answer the many questions that remain."
Last month, Sen. Elizabeth Warren accused Stumpf of "gutless leadership" and said he had not held himself accountable for the bank's actions. Before appearing on Capitol Hill, Stumpf told Jim Cramer on "Mad Money" that he would not resign.
The stock was last up more than 1 percent in extended trading.
The Consumer Financial Protection Bureau and Office of the Comptroller of the Currency declined to comment.
— CNBC's Ryan Ruggiero, Antonio Jose Vielma and Alex Crippen contributed to this report.