So while investors were told they were getting short-term liquid investments similar to a money market fund, according to the indictment they were instead funding Callahan's lavish lifestyle, and rather odd side investments.
(Read more: Busted! Inside one massive cybercrime ring)
Callahan cast a wide net for perspective clients. He allegedly received $600,000 from the Montauk volunteer fire department on Long Island. According to U.S. attorney, he told the firemen that their money would be invested in mutual funds and other securities but instead diverted the money to the Panoramic View and personal expenses.
To cover his tracks, prosecutors said, Callahan sent fraudulent account statements to the fire department that falsely showed that the investment was in 12 different mutual funds—but no mention of Panoramic View or the fact that Callahan and Manson each owned 50 percent of the development.
According to Richard Schoen, Montauk's fire chief, the department demanded its money back after the treasurer started asking Callahan questions about the whereabouts of the department's investment and was told by Callahan that the money was "moved around daily".
The scheme started to unravel when an independent auditor started to take a close look Callahan's various funds, authorities said. According to the indictment, Callahan and Manson provided bogus promissory notes representing debts owed to the Callahan funds. Callahan is also accused of sending the auditor documents with forged signatures.
The government is seeking forfeiture of the Panoramic property and any other property and goods obtained through the alleged Ponzi scheme.
(Read more: Behind this pump-and-dump scheme was...a chimp?)
Callahan's lawyer, Robert Knuts, has not returned CNBC's calls but Robert J. Anello, Manson's attorney said: "The criminal charges, as they relate to Mr. Manson, are unfounded. Mr. Manson consistently and expertly has acted consistent with the interests of the owners and investors of the Panoramic Property. For more than seven years, Mr. Manson has overseen the successful renovation and development of the beachfront historic property's luxury, full-service units and famous hotel. No allegations have been made that Mr. Manson had contact or involvement with any investors or that Mr. Manson made any investment decisions on behalf of the offshore investment funds or received any management fees based on those funds assets."
Anello Added, "Mr. Manson looks forward to a speedy resolution of the government actions, which will benefit its investors, unit owners, and guests."
Jeff Pohlman is a senior producer at CNBC. Andrea Day is a reporter at CNBC, covering crime and punishment.
You can watch all of our "Crime and Punishment" on CNBC's "Power Lunch" every Thursday, 1 p.m. ET.