Newman had been a portfolio manager at Diamondback Capital Management and Chiasson had held the same position at Level Global Investors. The original indictment alleged that the men were part of a "cohort of analysts" that traded on nonpublic information from employees at publicly trade tech companies.
The appeals court reversed the ruling on the basis that the two managers did not know where the inside information had come from.
Read MoreCommentary: It's time to legalize insider trading
"The Government presented no evidence that Newman and Chiasson knew that they were trading on information obtained from insiders in violation of those insiders' fiduciary duties," Judge Barrington Parker wrote in the filing.
Additionally the court emphasized that there was no evidence the people giving the tips did so for personal benefit, and the jury was improperly instructed as to this requirement, so the conviction of insider trading was invalid.
"We agree that the jury instruction was erroneous because we conclude that, in order to sustain a conviction for insider trading, the Government must prove beyond a reasonable doubt that the tippee knew that an insider disclosed confidential information and that he did so in exchange for a personal benefit," the judge wrote.
"Although the government might like the law to be different, nothing in the law requires a symmetry of information in the nation's securities markets," Parker added.
Read MoreLegalize insider trading? That's ridiculous: Andrew Stoltmann
Newman and Chiasson had been prosecuted by Bharara. After originally securing their convictions, Bharara said in a statement that the two joined "the ranks of high-level investment fund managers who are being made to answer for their extraordinarily bad risk-reward analysis about what is right and what is wrong."
A district court sentenced Newman to 4½ years in prison and Chiasson to 6½ years.
"Today's decision is a resounding victory for the rule of law and for Anthony Chiasson personally," Chiasson's lawyer Gregory Morvillo said in a statement to CNBC. "Mr. Chiasson has always conducted himself according to the highest ethical and professional standards in service to many of the world's leading hedge fund investors who were his clients for years."
Newman's lawyers, Stephen Fishbein and John Nathanson, said in a statement they "are relieved but not surprised by today's decision."
"Unfortunately, this vindication comes after four years of unnecessary prosecution including a trial in which the Second Circuit held that the wrong legal standard was applied," they said. "We are gratified that, going forward, others will benefit from clearer rules in this area."
The case had been closely watched among the white collar defense lawyers for whether the court would impose a heightened standard for proof in insider trading cases.
Read MoreThe truth about insider trading
It marked a blow for Bharara, who had secured convictions of 82 other people in insider trading cases since October 2009, with one trial acquittal but never before an appellate reversal.
Bharara said in a statement that the ruling "interprets the securities laws in a way that will limit the ability to prosecute people who trade on leaked inside information."
As a consequence, he said, the court's decision will affect "a subset" of recent cases, but he insisted his team "investigated and prosecuted misconduct based on our good faith assessment and understanding of the facts and the law that existed at the time."
The U.S. attorney said his team is still analyzing the decision, adding that it "appears in our view to narrow what has constituted illegal insider trading."