While the bond-buying program has served a purpose, the Fed official compared the Fed's $85 billion in monthly asset purchases to a narcotic — "monetary ritalin" — warning that there's a risk the markets and the economy become too addicted.
Fisher has been a critic of the Fed's easy monetary policy warning that it won't do much to support job creation and may actually hurt the economic recovery.
The economy is improving, Fisher said, but job creation isn't picking up fast enough.
On Wednesday, Fed Chairman Ben Bernanke told Congress that it may take another three years before the unemployment rate reaches 6 percent.
(Read More: Fed's Bernanke Stays the Course)
"We believe the monetary policies that we've conducted have helped get a stronger recovery and more jobs than we otherwise would have had," he said during his semiannual Congressional testimony.
With the jobless rate at 7.9 percent in January, Bernanke said his "reasonable guess" would be that it will take three more years before the unemployment rate reaches 6 percent. Late last year, the Federal Reserve said it would keep interest rates low until unemployment reached about 6.5 percent.
Fisher attributed this sluggish job growth to fiscal uncertainty. "I argue it's not because of monetary policy. We provide a lot of fuel," Fisher told CNBC. "It's because of the uncertainty and the fog that's been created by the fiscal authorities because people don't know what their tax bill is going to be, what their costs are going to be."
With the real issue fiscal policy uncertainty, Fisher said, "If anything now, with the housing market back, with the market trading these large volumes, I think it's good for us to at least socialize the concept that we may taper off — not quit entirely — the rate of purchase. At least that's what I'm arguing at the table but I'm in the minority."