The breakthroughs—previously undisclosed—show the promise of the agency's 20-month-old whistle-blower program.
Yet, the program faces challenges on many fronts.
Some Wall Street firms are urging employees to report wrongdoing internally before running to the government, and one hedge fund, Paradigm Capital Management, was accused in a lawsuit of punishing an employee who had cooperated with the S.E.C., according to court and internal documents.
Another financial firm, the documents show, pressured an employee to forfeit potential "bounties or awards"—a possible violation of S.E.C. rules.
Some lawyers also complain that the agency takes weeks or months before it responds to a backlog of tips, while others question whether the S.E.C. overstates the power of the whistle-blower program.
More From the New York Times:
Regulators to Restrict Big Banks' Payday Lending
Health Provider Strives to Keep Hospital Beds Empty
Galaxy S4 Puts More Features Into the Same Package
"The S.E.C.'s program hangs in the balance right now," said David K. Colapinto, general counsel of the National Whistleblowers Center, a nonprofit group. "Everybody is waiting to see what they do—and nobody really knows what the track record of this program is going to be."
For its part, the S.E.C. notes that it fields dozens of tips a week. To cope with the onslaught, the agency has ramped up its whistle-blower team to include 11 lawyers and 3 paralegals.
"You'd be hard pressed to find another whistle-blower program that's quicker," said Sean McKessy, the director of the program.
And Mr. McKessy's office has quietly gained momentum. The investigation into Knight Capital stems from last summer, when the firm introduced new trading software.
The firm bungled the switch and sent a flood of erroneous orders to buy shares. Knight later had to sell the stocks at a loss of more than $4oo million, draining capital and pushing it to the brink of collapse.
The S.E.C. initially charted a narrow investigation, lawyers briefed on the case said. But when a whistle-blower came forward, the lawyers said, the agency was able to shift gears and expand the investigation.
Regulators are looking at whether the firm broke rules that govern which customers and traders can gain direct access to stock exchanges.
Knight, rescued by a group of investors, has not been accused of any wrongdoing. It declined to comment.
A whistle-blower also played a role in the recent investigation of the investment firm Oppenheimer, the lawyers said.
With the tipster's help, the S.E.C. fined the company nearly $3 million last month for inflating returns in one of its private equity funds.
The agency had another breakthrough after pursuing the China Voice Holding Corp. for more than a year, which was suspected of operating a growing Ponzi scheme.
The turning point came when Dee Dee Stone, an outside consultant preparing the Dallas company's tax return, produced documents that laid bare a scheme hidden in the company's books.
Without naming the consultant or China Voice, the S.E.C. announced in August that it had doled out its first whistle-blower award. And Ms. Stone, 38, who collected about $46,000 to date, said in a recent interview that "I just wanted to stop the scheme before the investors lost everything they worked for."
The S.E.C.'s whistle-blower program was born from painful missteps. For years, the agency ignored tips about Mr. Madoff's glowing returns and rewarded tipsters only if they alerted the S.E.C. to insider trading cases.
Under pressure to change, the agency inaugurated its whistle-blower office in August 2011. The agency hired Mr. McKessy, a former executive at AOL and Caterpillar, to run the office.
Under the program, tipsters can reap up to 30 percent of the money the S.E.C. collects when fining a company or its executives.
To qualify for the reward, a whistle-blower must turn over new information that leads to successful enforcement actions yielding more than $1 million in fines.
With the corporate world grumbling, the S.E.C agreed to consider dishing out some of the highest possible awards to employees who report fraud at work before turning to the government.
Tipsters appear to have taken the S.E.C. up on the offer; more than 80 percent of aspiring whistle-blowers have reported internally first, according to people briefed on the matter.
Private lawyers also seized the opportunity. Stuart D. Meissner, a New York area lawyer, bought the Web site SECsnitch.com, and solicited tipsters with advertisements in theaters before the movie "Wall Street: Money Never Sleeps."
Calls from whistle-blowers came pouring in. In a recent report to Congress, Mr. McKessy stated that his office in the last fiscal year received 3,001 tips.
"I expect you'll see more cases in the coming year," said Jordan A. Thomas, a former S.E.C. official who helped create the office and is now handling whistle-blower complaints at the law firm Labaton Sucharow.
The S.E.C. has boasted about the "high quality" of the tips, saying that several each week have proved fruitful.
But some S.E.C. officials have privately acknowledged that those numbers are somewhat exaggerated. And whistle-blower lawyers have complained about waiting months between inquiries.
The agency has also encountered obstacles from companies.
In recent months, several banks drafted policies compelling their staffs to report fraud internally, lawyers say.
Some companies require employees to attest annually that they never witnessed any fraud, a certification that could be used to discredit employees who later blew the whistle.
Corporate lawyers say the new policies help root out fraud at an early stage. "There is a sharper focus to encourage employees to keep their complaints at home through positive steps, rather than through punitive measures," said Lloyd B. Chinn, a partner at Proskauer and co-head of the law firm's whistle-blowing and retaliation group.
But whistle-blower lawyers say that some policies violate the spirit of the S.E.C.'s program.
David J. Marshall, a founder of the whistle-blower law firm Katz, Marshall & Banks, said that he had seen companies subtly insert language into severance agreements that looks innocuous, but that effectively stops the employee from going to the S.E.C.
In a few cases, he said, companies have asked departing employees to sign a document certifying that they have told the company about any confidential information the employee has given to outside parties.
"The craft is in designing language that doesn't say that, but that can have that effect," Mr. Marshall said.
In other cases, companies are accused of retaliating against whistle-blowers, in violation of S.E.C. rules.
James Nordgaard, a hedge fund trader, said in a lawsuit that his employer, Paradigm Capital, "embarked on a campaign of retaliation" against him, after he reported "trading violations" and a "deliberate disregard for compliance rules." Mr. Nordgaard claims he was removed from the trading desk, isolated in a "room with little ventilation" and later "demoted."
In reply, Paradigm wanted to move the case before an arbitration panel. Mr. Nordgaard agreed to do so.
Despite the challenges for the S.E.C., praise for the program has come from one of the agency's longtime critics: Harry M. Markopolos, a securities analyst who suspected the Madoff fraud a decade ago, drew it to the S.E.C.'s attention, and was ignored.
In an interview, Mr. Markopolos said that when he has approached the new whistle-blower office, "they are very thankful and they actually do investigations."
But Mr. Markopolos said that the agency has yet to bring the kind of big cases that would prove the worth of the new program.
"The question is whether they can turn it into successful enforcement actions," he said.