"The sooner we come to grips with this excessive level of assets on the balance sheet of the Federal Reserve—that everybody agrees is excessive—the better," he said in a "Squawk Box" interview. "There is a general presumption that we can wait indefinitely and make judgments on when we're going to move. I'm not sure the market will allow us to do that."
But if the Fed moves too quickly in reining in its accommodative policies, he added, it could shock the market, which is already dealing with a very large element of uncertainty.
Greenspan said he's not sure the markets will allow an easy exit. "Gradual is adequate, but we've got to get moving."
(Read More: Markets May Have Gone Too Far on Taper Talk: Plosser)
"The most important positive force in the economy at the moment is the fact that equity premiums are so high, which means the downside on stock prices is quite limited," he said. "If we can get stock prices to rise, which they will if this thing stabilizes, then you get a lot of asset-growth effect on the economy."
"I think the issue is not only a question of when we taper down, but when do we turn," he explained, meaning actually decreasing the Fed's balance sheet, which stood at $3.357 trillion on June 5, compared with $3.342 trillion on May 29.
"The markets may not give us all the leeway we might like to do that," he observed, pointing out that tapering is still increasing the Fed's balance sheet.