Read MoreBanking culture promotes cheating: Research
John Rogers, former CEO of the CFA Institute and previously a top Invesco executive, agreed.
"Making senior management personally responsible resonates with me. That's true for actors down inside the banks too. They need to be apprehended, tried and punished," Rogers explained, speaking alongside Ellis.
Rogers added that misbehavior would happen again, making existing oversight mechanisms important.
"To think that the banking system is ever going to have just white hats on is unfortunately not going to happen. That's why there are safety nets in place to help deal with that," he said. The safety nets that Rogers referred to include federal oversight put in place after the financial crisis, including higher cash reserve requirements for banks.
While Ellis took a harder tone, he did note that most people on Wall Street are ethical. "It's a very small fraction of people, and usually with a very small fraction of their time, that did things that were really crummy," he said.
Still, Ellis was highly critical of certain behavior before the financial crisis. "We have had some real SOBs," he said.
Ellis gave the examples of former Lehman Brothers CEO Richard Fuld, who now acts as a deal consultant at a small firm he founded, Matrix Advisors in Manhattan.
He also called the "former CEO of Merrill Lynch"—presumably Stan O'Neal—"probably one of the most despicably behaved human beings we've ever seen in this country."
O'Neal famously left Merrill Lynch with $161 million in accumulated benefits as the bank fell into crisis. O'Neal has largely disappeared from public view besides serving on the board of metal company Alcoa. (The next CEO was John Thain; Ellis later declined to clarify which executive he meant.)
"You think, what penalties have they paid? Maybe a little embarrassment with their kids' friends who all know what happened, sort of," Ellis said. "Most of it has been a free pass."
Read MoreCharley Ellis wants to rescue your retirement
Ellis also mentioned former Goldman Sachs trader Fabrice Tourre, who was punished. Tourre was found liable for defrauding investors in mortgage-backed securities; he paid a fine and is now pursuing a doctorate in economics at the University of Chicago. Ellis said if Tourre's supervisors had been more afraid of personal penalties, they would likely have spotted and prevented his actions.
A spokesman for Bank of America Merrill Lynch declined to comment. A spokesman for Goldman Sachs did not respond to a request. Fuld did not respond to a request for comment left with his assistant at Matrix. Tourre did not respond to an email. Attempts to contact O'Neal were unsuccessful.