Wells Fargo was recently fined for creating accounts for customers across multiple product lines without telling them, in order to meet sales goals. All told, the bank agreed to pay $185 million in penalties and $5 million to customers. Over the course of five years, 5,300 employees were also fired.
Armstrong also believes the auditors failed in not detecting the problem soon enough.
"Where were they? They should have been raising some red flags on these issues and they should have seen it and warned the board of directors and management much sooner," he said.
When asked for a comment on Armstrong's views, a spokesperson for Wells Fargo referred to Stump's interview with CNBC's "Mad Money" last week.
"Of course it stops with all of us, and especially me. I'm the leader, I get it. I said right off the bat when we don't meet our goals with 100 percent right, I'm accountable," he told Jim Cramer last Tuesday.
"And I'm leading this company and leading it forward through this. I want to make sure that every customer knows when they come into one of our branches, deal with one of us on the internet, whatever the case is, we are 100 percent on their side."
Stumpf also said he does not plan to resign.
"I think the best thing I could do right now is lead this company, and lead this company forward," he said.
The CEO has been called to testify Tuesday in front of the Senate Banking Committee. Wells Fargo is also under investigation by the House Financial Services Committee and is being sued for fraud and negligence by customers.
— CNBC's Laura Petti and Abigail Stevenson contributed to this report.