Damage to city budgets: Some was self-inflicted

Five years after a once-in-a-lifetime global financial collapse and the Great Recession, many American cities now caught in a financial bind can rightly blame forces beyond their control.

In some cases, though, the damage was self-inflicted.

Nowhere is that more apparent than the ongoing budget drain from hundreds of billions of dollars' worth of failed bets known as interest rate swaps. These complex financial derivatives—highly profitable for the investment bankers on the other side of the deal—were supposed to help cities lower the cost of selling municipal bonds to finance everything from new schools to underfunded pensions.

Instead, these deals have turned into municipal bombs with long fuses. City officials now digging through the ashes of these failed financings have learned the hard way that—in many cases—Wall Street's wizardry cost taxpayers much more than the plain vanilla, fixed-rate bonds that have funded public projects for generations.

"The banks made five times more money on the swaps than (they) did on the underwriting of the bonds—it was a gold mine," said Andrew Kalotay, a public finance consultant who advises local governments. "The transaction costs upfront were horrible, but (local officials) didn't understand that."

Oakland, Calif. was among the cities taken for a costly swaps ride by Wall Street, according to Rev. Curtis Robinson, a local pastor who spent two decades working in the financial services industry before joining the clergy.

"Oakland was just outgunned," he said. "You have to send the right kind of people to do those kinds of deals... They just didn't understand it. They were just very, very short term on their perspective."

Oakland's dubious deal was typical of hundreds of swaps that are now draining money from local governments as they scramble to make up deep shortfalls in tax revenues.

In 1997, to try to lower the cost of covering city workers' pension benefits, Oakland officials decided to refinance a fixed-rate bond with a new one that paid investors a floating return, based on a benchmark interest rate. Cities selling floating rate bonds try to borrow more cheaply, much the way a home buyer can borrow more cheaply with an adjustable rate mortgage.

To hedge the risk that higher interest rates might drive up future floating payments to investors, Oakland also bought an interest rate swap from Goldman Sachs. That instrument requires the bank to cover Oakland's floating rate payments to investors. In return, Oakland agreed to pay Goldman Sachs a fixed payment—of 5.7 percent—until 2021.

That turned out to be a very bad deal for Oakland.

"It was awhile back, when the quote-unquote experts knew that interest rates weren't going any lower that they did the deals," said Robert Brooks, a University of Alabama finance professor and an expert on municipal finance reform. "'Let's fix the rate now because we all know 5 percent is as low as it's ever going to go.' Well, now they wake up and it's two (percent). It was clearly a casino bet—and it was promoted by Wall Street."

Wall Street had good reason to promote highly profitable swaps deals like Oakland's. And before the bottom fell out of the financials system, it was fairly easy to sell them as a sure bet to local governments.

Derivatives like Oakland's interest rate swap are cousins of the complex securities that were supposed to turn subprime home mortgages into gold. But much like the dodgy paper that brought down the U.S. housing market and financial system, these municipal bond bets backfired in 2008 when municipal bond insurers—swamped by bad mortgage bet—went bust.

That meant Oakland, and thousands of other cities that relied on insurers to back their floating rate bonds, was forced to refinance its floating debt with new, fixed-rate bonds. As a result, the city is now stuck with a swap that's hedging an interest rate risk the city no longer has. In Oakland's case, it also means paying Goldman Sachs $4 million a year for the swap—or coming up with roughly $16 million to get out of the deal altogether.

Oakland is not alone. Chicago, Denver, Houston, Philadelphia, Massachusetts, New Jersey, New York, California and Oregon are among dozens of states, cities, school districts and other public entities that are now holding money-losing swaps, according to a spot check of city financial statements by Service Employees International Union, which represents more than a million state and local government workers.

Many of the officials who signed off on these swaps are long gone. But across the country, city hall incumbents are under heavy political pressure to unwind these deals.

(Read more: Issue municipal bonds? No thanks, we'll crowdfund instead)

"The spreads (investment banks) are making is just unbelievable," said Bahar Tohou, a researcher for SEIU. "And they're making that profit at a time when we have to close schools, increase class size and close fire stations and hospitals and libraries. If we can renegotiate these deals by even half a percent or one percent we can save taxpayers millions of dollars."

But bankers holding city officials to these swaps argue that the deals did just what they were supposed to do, and that most cities saved their taxpayers money before the collapse. They also maintain that—much like a borrower with adjustable rate mortgage who complains when rates rise—cities with losing swaps bets that made money when rates were higher can't expect to have it both ways.

Oakland, for example, saved some $37 million on borrowing costs during the life of the swap, according to a consultant hired by the city last year. The consultant's advice: See if you can get Goldman Sachs to renegotiate the deal.

Goldman Sachs declined to comment on the bank's negotiations with Oakland. But CEO Lloyd Blankfein was asked about the Oakland swap deal at the company's annual shareholders meeting in May, 2012.

"If we are at the lowest level of interest rates today, that means that (for) every fixed loan that happened before this, it would be advantageous to the borrower to tear that up and re-borrow today when interest rates are lower," he said. "That's not how the financial system could work…I don't think it's a fair thing to ask."

The fairer question, say critics, is why Wall Street sold Main Street on the idea of putting taxpayer's money into such a risky investment in the first place—and whether those risks were fully disclosed.

"Why were investment banks pitching ideas to municipalities that they themselves would not touch?" said University of Alabama's Brooks. "The (banks) weren't issuing variable rate debt and swapping it for fixed. They wouldn't touch that deal."

(Read more: Lehman sues Giants Stadium over interest-rate swaps)

Proponents of municipal finance reform like Brooks argue the problem goes far beyond failed rate swaps. Too often public financings are sold to local officials based on Wall Street's opportunity to make money, he said, instead of the financial needs of the local municipality.

"Financial derivatives in the municipal space are sold—not bought," he said. "Investment bankers fly into the county and pitch an idea. The problem with that is you don't have someone on the county side of the table who can vet the idea. Typically elected municipal officials and their politically appointed staff are just not capable of doing that analysis."

Ironically, local officials now trying to pay off bad swaps may be getting their analysis wrong all over again, according to municipal finance advisors. Closing out a swap that's losing money when interest rates are near zero—just when rates soon may be rising again—could end up throwing good taxpayer money after bad. If rates do get up off the floor, cities that stand pat on their upside-down swaps bets could reduce the long-term cost of their swaps over the next decade.

But that's a tough sell to taxpayers angered about big payments going to Wall Street while budget cuts elsewhere force teacher layoffs or city pool closings.

(Read more:America's cities on the edge)

In response, a few cities have taken their Wall Street bankers to court—claiming the swaps game was rigged against them. That legal argument got easier to make after U.S. and European regulators last year uncovered a massive scheme to rig a benchmark interest rate, known as Libor, that's widely used in setting swap payments. In July, Sacramento County joined other counties and cities - including Houston, Baltimore and Philadelphia - suing their bankers for damages, arguing that they're paying more on their swaps deals than they should because the Libor rate was rigged.

So far, American and British regulators have fined UBS, RBS, Barclays, Rabobank and ICAP $3.7 billion in connection with rate rigging. Seven individuals face criminal charges. European regulators are reportedly considering an additional $2 billion in fines for banks involved in fraud. The case is being watched closely by cities still holding onto their money-losing swaps.

But suing a big bank is a costly undertaking that could come back to haunt local officials when it comes time to float the next bond offering. Since municipal financing is all but impossible to arrange without the help of a bank to underwrite the offering and manage the sale of the bonds, local officials have a weak hand when they try to renegotiate soured swaps bets.

"Many people said we shouldn't repay the loan," said Oakland Mayor Jean Quan. "Unfortunately if I don't repay that loan, in a city that's bonded heavily because we're growing, and we're building stadiums, it will be suicide. And it would cost us so much more."

By CNBC's John W. Schoen. Read him on Twitter @johnwschoen.