Earlier in the day, Stumpf testified in front of the Senate Banking Committee about the opening of authorized accounts for customers. He apologized and said he was committed to fixing the problem.
But Vitter pointed out that Stumpf said he didn't know about the fraud until a couple of years after it occurred.
"This was widespread fraud over many years that according to his testimony today, he and the board didn't know about for several years. Something's wrong there," he said in an interview with CNBC's "Closing Bell."
At the hearing Tuesday, Sen. Elizabeth Warren called for the resignation of Stumpf. Vitter said that while he doesn't think Congress should "micromanage any business," the case certainly raises concerns about the ability for anyone to properly manage or regulate "these mega-institutions."
"Maybe these institutions, we talk about too big to fail, they seem like they are too big to manage or too big to properly regulate also," he said. "That's a trendline in our banking industry that I think is very dangerous."
Wells Fargo was recently fined for creating accounts for customers across multiple product lines without telling them, in order to meet sales goals. All told, the bank agreed to pay $185 million in penalties and $5 million to customers. Over the course of five years, 5,300 employees were also fired.