Billionaire hedge fund manager Paul Singer holds a pessimistic view of the global financial system, saying he believes the Federal Reserve easing back on its asset-purchasing program is "off the table" even though investor confidence could sour quickly.
"The conditions for a loss of confidence are here right now," Singer said at The Wall Street Journal's Heard on the Street Live event in Manhattan. Singer runs $22 billion Elliott Management, one of the oldest and most successful hedge funds in the world.
Singer said Janet Yellen—the likely successor to Ben Bernanke as Federal Reserve chair--would be unlikely to end the country's unhealthy dependence on debt the way Paul Volcker did in the same position during the 1970s and 1980s.
"There's an absence of any signs that political leaders are willing to face this problem and address and restructure these obligations," Singer said.
If a reduction in the Fed's $85 billion a month bond-buying program ever starts, Singer said it was "very, very uncertain" what would happen.
Regardless, Singer said it will "matter tremendously" if investors stop believing in the economy.
"Confidence could be lost if some combination of a rise in the price of commodities or gold, the consciousness of inflationary potential, a continued lack of accelerating growth, social unrest, an acceleration of quantitative easing," Singer said. "It will matter."
"The s--- hits the fan when markets wake up and focus on it," Singer added in response to a question about when debt will become too much of an economic burden. "No growth rate and tax rate can fix this. It needs to be fixed by restructuring. There's no inflating out of it."
Singer said that while most investors are doing well financially, the average American is still in a difficult situation given high unemployment and other uncertainties.
"Shouldn't we be pivoting to more sound, pro-growth policies?" he asked.
On a micro level, Singer said his longstanding battle with Argentina over its debt payments wasn't likely to end soon given news Monday that the U.S. Supreme Court decided not to hear a related case.
(Read more: )
Singer offered few specific investment recommendations.
He said he understood the case for growth in China but that he was "frequently surprised and disappointed" by investments there despite considerable due diligence.
—By CNBC's Lawrence Delevingne. Follow him on Twitter @ldelevingne.
This story has been updated to correct Paul Singer's name.