"It's been a hot industry, probably a little too hot," said Dick Evans, chief executive of Cullen/Frost Bankers of Texas, which has a relatively sizable energy practice. "But it is not time to panic. We have been in the game a long time. I am comfortable with what we have been doing."
There is a flip side to lower oil prices that helps the banks, or at least those with large consumer businesses. The less cash consumers have to spend filling up their gas tanks or heating their homes, the more emboldened they may feel to sign up for a credit card or take out a mortgage.
Read MoreGas slump fuels tax hike talk
"As consumers have more money in their pocket, surely that helps Wells Fargo," the chief executive of that bank, John G. Stumpf, said at a financial services conference last month. "I would say net-net this is a good thing for the country."
Still, if oil prices remain near $50 a barrel for long, economists and industry analysts expect a sharp deceleration in production this year, idling energy bankers and cutting into their lucrative fees.
Two of the banks that may be the hardest hit by lower investment-banking fees are among the biggest. Wells Fargo derived about 15 percent of its investment banking fee revenue last year from the oil and gas industry, while at Citigroup, the business accounted for roughly 12 percent, according to the data provider Dealogic.
At some of the larger banks in Canada, a slowdown in fees could be even more pronounced. At Scotiabank, about 35 percent of its investment banking revenue came from oil and gas companies last year.
And Wall Street firms that financed energy deals may now have trouble offloading some of the debt, as they had originally planned.
Morgan Stanley, for instance, led a group of banks that made $850 million of loans to Vine Oil and Gas, an affiliate of Blackstone, a private equity firm. Morgan Stanley is still trying to sell the debt, according to a person briefed on the transaction. Similarly, Goldman Sachs and UBS led a $220 million loan last year to the private equity firm Apollo Global Management to buy Express Energy Services. Not all the debt has been sold to other investors, according to people briefed on the transaction.
A precipitous drop in oil prices can quickly turn loans that once seemed safe and conservatively underwritten into risky assets.
The collateral underpinning many energy loans, for example, is oil that was valued at $80 a barrel at the time the loans were made. As oil has dropped well below that price in recent months, the value of the bank's collateral has sunk.
Many oil companies have bought hedges on oil prices, which are providing lenders with additional cushion. But when those hedges expire, and if oil prices remain low, the banks may need to reserve money against the loans.
"At $50 a barrel, things can get a bit testy," said Christopher Mutascio, a banking analyst with Keefe, Bruyette & Woods.
Some of the greater risks may be the loans the banks have extended to the many kinds of services companies that work in and around the oil industry. Some of these services companies, lured by the boom, may have short track records, analysts say.
Low oil prices can have ripple effects that many banks may not anticipate, particularly in states such as North Dakota and Oklahoma where energy is a large driver of the economy.
When oil prices crashed in the 1980s, many Texas banks failed not because of loans to oil producers, but because of loans to local real estate developers who had been caught in the energy bust.
Read MoreOil sector in dire straits: Schork
Just over 20 percent of the loans at MidSouth Bank, based in Lafayette, La., are to oil and gas companies, a high proportion relative to its peers. But Rusty Cloutier, MidSouth's chief executive, said the bank had focused its lending on services companies with seasoned management that were most likely prepared for a dip in activity.