The company's board of directors elected President and Chief Operating Officer Tim Sloan to succeed him as CEO, while Lead Director Stephen Sanger will serve as the board's non-executive chairman.
The news comes after it was revealed that employees in Wells Fargo's community banking division opened about 2 million accounts without customer authorization, which resulted in the bank paying $185 million in penalties. Stumpf was grilled on Capitol Hill as he defended the bank's sales practices.
Sloan told CNBC late Wednesday that Stumpf "concluded this was the right thing to do for the company to allow it to move forward," but added the work is far from finished.
"There's a lot to be done. No one individual is necessarily going to make the difference," he said. "We all have to work together to restore the reputation of the company to continue to focus on moving the company forward."
Stumpf said in a statement that he was "grateful for the opportunity to have led Wells Fargo" and is optimistic about the bank's future.
"While I have been deeply committed and focused on managing the Company through this period, I have decided it is best for the Company that I step aside," he said in a statement.