The 2nd Circuit returned the case to Rakoff, who must review the settlement again unless he pursues a further appeal. One judge on the panel, Raymond Lohier, said he would have ordered Rakoff to approve the accord.
Rakoff, in an email, said: "I don't think it's appropriate to comment."
Citigroup Spokeswoman Danielle Romero-Apsilos declined to comment. The SEC had no immediate comment.
Read MoreJudge to SEC: No More 'Settlements'
Wednesday's decision is a rebuke to Rakoff, who in two decades on the bench has earned a reputation as a brilliant, iconoclastic judge and whose voiding of the Citigroup accord in November of 2011 prompted other judges to scrutinize similar settlements.
Rakoff has also been a thorn for the SEC, having in 2009 rejected as "a slap on the wrist" its original accord with Bank of America over the bank's purchase of Merrill Lynch.
His decisions have spurred a debate over the proper SEC role in policing Wall Street and have been followed by big changes at the regulator.
Read More SEC, FBI probe Icahn, Mickelson over insider trades: Report
Last June, SEC Chair Mary Jo White said the regulator would begin requiring admissions of wrongdoing as a condition of settling particularly serious civil charges, citing the need for accountability to the public. White is a former federal prosecutor who joined the regulator last year.
Citigroup's settlement was meant to resolve claims that the bank misled investors in 2007 about a $1 billion collateralized debt obligation, Class V Funding III, and simultaneously bet against the debt as the U.S. housing market began to falter.
Rakoff rejected the accord as neither fair, nor adequate, nor in the public interest, saying the "neither admit nor deny" policy left him no facts on which to judge the settlement.
Read More SEC probe found accounts at Merrill, Schwab linked to Mexican drug cartels: Reuters
Pooler, however, said such fact-finding should be left for a trial, while "consent decrees are primarily about pragmatism."
The SEC did go to trial against former Citigroup manager Brian Stoker, the only individual sued over the CDO, but a jury in July 2012 found him not liable.
In Wednesday's decision, the 2nd Circuit also adopted a new standard for reviewing settlements with U.S. enforcement agencies that involve injunctive relief, saying judges must assess whether such accords are fair and reasonable and whether the public interest "would not be disserved."
Read MoreNew York Stock Exchange to pay $4.5M to settle SEC charges