A three-judge panel of the court said in a written decision that there was sufficient argument that Madoff, 70, was a flight risk, given his age and the possibility that he could be sent to prison for the rest of his life at sentencing.
He faces up to 150 years—effectively a life sentence.
The judges said Madoff, who had been under house arrest in New York until his guilty plea last week, could also have a secret stash of money abroad to finance a potential escape.
"The defendant has a residence abroad, and has had ample opportunity over a long period of time to secret substantial resources outside the country," the judges of the U.S. Court of Appeals for the Second Circuit wrote.
Madoff owns a home in France.
Madoff attorney Ira Lee Sorkin told CNBC, "We are disappointed, and we respectfully disagree with the Court's decision, but the Court has ruled." He would not comment on plans for any further appeals.
While his attorneys argued his assets have been frozen and he has made no effort to flee, the appeals court said the U.S. District Court judge presiding over the money manager's criminal case "was not required to treat this defendant's financial representations as reliable."
The decision means Madoff might never again see his luxury Manhattan penthouse where he had been under home confinement for three months until last week's courtroom confession.
He is scheduled to be sentenced on June 16.
Madoff's attorneys had sought to allow him to remain out of jail on bail until then, as is often allowed for white-collar criminal defendants who are not considered violent or risks of flight.
Madoff could potentially seek a rehearing before the full appeals court, but such a bid is considered a long shot and there would be no guarantee the full court would agree to even hear the case.
The ruling came a day after the appeals court held oral arguments on Madoff's bail appeal.