They submit reports that run into the thousands of pages to regulators who then make their own estimates for how banks would perform using Federal Reserve models. If the Fed does not like what it sees, the bank cannot increase dividends to shareholders or buy back more stock.
A wide gap
What irks banks is that their stress test results and the Fed's are often far apart, and regulators give little information about how or why they disagree.
For instance, in 2014 Zions Bancorp, a Salt Lake City, Utah-based bank with $55 billion of assets, said its assessment showed it passed the stress tests: in a hypothetical downturn, it would have capital equal to about 5.9 percent of its assets, above the Fed's 5.0 percent threshold. But the Fed ruled Zions's capital ratio would fall to 3.6 percent in its scenario and issued a failing grade.
Executives also complain that strong capital levels are often not enough: well-capitalized banks can still be prohibited from paying out more dividends for "qualitative" reasons, such as a flaw in its capital planning process. Banks includingCitigroup, BB&T, and Ally Financial have failed the qualitative portion of the stress tests in recent years.
To combat the opacity, banks have met in forums hosted by the Clearing House, a trade group for big U.S. commercial and retail banks, the American Bankers' Association, and at other informal venues.
Read MoreWhere to put your cash? A house or a stock
Among the problems they have discussed is how to effectively write their reports, which must contain detailed information on the mathematical models they employ as well as a narrative piece that describes a lot of the methods that banks used to arrive at their conclusions. Executives say it is a challenge to produce a document that clearly and concisely explains everything to Fed modelers and examiners who may be unfamiliar with individual institutions.
One senior executive at a large U.S. bank involved in the stress-test process said one thing the Fed pays close attention to is the changes a bank made since the last submission. Instead of describing those changes in an appendix, the executive has learned the Fed preferred it to be addressed at the beginning of the document.
The executive added that given that the Fed has a limited period of time to review submissions, examiners have told other banks that they prefer to see a more detailed executive summary.
The Fed itself has taken some steps to help banks get a better sense of its expectations on the qualitative portion of the stress tests and how banks should organize their information—its October instructions for the 2015 stress test included a section on their preferential format.
The instructions also contained a "common themes" appendix that explained issues the Fed was looking closely at, such as if assumptions banks used to estimate losses are clearly articulated and how banks account for potential limitations and weaknesses in the models they use.
But some executives say that the more specific information about what it is looking for is not helpful if the Fed does not detail how it reached its conclusions.
"We've had a lot of conversations about methodologies and the approach to transparency in the CCAR process that have been more of a concern," said Chris Halmy, finance chief of Ally Financial, in an October interview. Ally has been seeking clarity "less about instructions and more about the feedback," Halmy said.
Another unresolved issue that banks have been powwowing about is how to estimate legal losses nine quarters into the future. In its common themes appendix, the Fed faulted many banks' projections and said they needed to take into account "possible claims of all types."
Legal losses are notoriously difficult to forecast. Without any more specifics about what the Fed wants banks to incorporate into their forecasts, it is a struggle to produce a result that will satisfy regulators, said one consultant that does stress-testing work for large banks.
"No one really knows what the Fed is looking for" with respect to potential legal losses, the consultant said. "There may be some objections to capital plans next year as a result."