The luncheon featured dining on chicken and salmon and had an atmosphere that was "collegial, probably a little too collegial," according to someone quoted by the Journal.
Read MoreBankers should be rewarded for avoiding risks: Tarullo
Ethical transgressions are certainly nothing new to Wall Street, but the crisis and its tales of bankers' taking advantage of consumers and investors—"muppets," as one Goldman trader infamously referred to them—have brought greater urgency to the issue.
Bankers reportedly have talked about setting up an ethical standards board along the lines of what exists in the U.K., but Wall Street has balked.
Things need to change soon, Dudley warned.
Read MoreNow Steve Cohen is paying for being extra ethical
"A number of factors have contributed to the cultural failures that we have seen. An important question is whether the sheer size, complexity and global scope of large financial firms today have left them 'too big to manage,' " he said.
"Financial firms exist, in part, to benefit the public, not simply their shareholders, employees and corporate clients. Unless the financial industry can rebuild the public trust, it cannot effectively perform its essential functions," he added. "For this reason alone, the industry must do much better."
Dudley suggested compensation "can play a useful role in reducing excessive risk taking," an idea floated recently by the International Monetary Fund, which said pay should be tied to responsible risk management.
Read MoreIMF: Time has come for big changes in banker pay
Failure to implement reforms would have dire consequences, he added.
"In conclusion, if those of you here today as stewards of these large financial institutions do not do your part in pushing forcefully for change across the industry, then bad behavior will undoubtedly persist," Dudley said. "If that were to occur, the inevitable conclusion will be reached that your firms are too big and complex to manage effectively. In that case, financial stability concerns would dictate that your firms need to be dramatically downsized and simplified so they can be managed effectively."