Read MoreA CEO in cuffs? 2014 Wall Street crime predictions
Bharara's office has already indicted at least one significant player: SAC Capital, the Stamford, Conn., hedge fund that pleaded guilty late last year to securities fraud after a rash of current and former employees were charged with insider trading.
More recently, Bharara and his colleagues at the Justice Department have tangled with JPMorgan Chase, which in November paid a record $13 billion to settle charges that it misled investors about the quality of certain mortgages. Other cases against the large bank are still pending.
On many occasions, Bharara said, Wall Street firms being targeted by his office in investigations argue that terrible consequences will follow if tough sanctions are levied—a huge drop in the stock, for instance, or executives walking away from their leadership roles.
"What I have found typically is that, in reality ... the sky does not fall," Bharara said. "In fact, sometimes the sky brightens: stock prices remain steady, or go up, as the company is viewed as putting problems behind it, clients and customers and key employees don't even bat an eye, and sometimes, the CEO even gets a raise."
Read MoreNo one is too big to jail, Wall Street cop says
"And so," he added, "this repeated chicken little routine ... begins to wear thin."
Bharara's reference to rising stock prices and CEOs getting pay increases may have been an allusion to JPMorgan. The company's stock has risen 28 percent in the past year while its CEO, Jamie Dimon, received a large raise for his performance in 2013, despite the mortgage-fraud settlement and a confluence of other high-profile cases.
A JPMorgan spokesman declined to respond to whether the comments were a dig. But an analyst who covers the bank dismissed Bharara's comments as toothless. "If that's what he wanted," the analyst said, referring to the idea of a JPMorgan indictment, "why didn't he stipulate that? What is he actually trying to accomplish?"