"I think the real issue is that the Fed has expanded its tool kit so dramatically, and really there are some real questions as to how potentially it unwinds, when it unwinds," he said. "We saw this past summer there was this announcement of potentially a taper and the markets actually tanked, and after that the Fed backpedaled. What's going to happen if we go on for months, years longer?"
Huszar apologized for his role in QE in a Wall Street Journal op-ed published Monday.
(Read more: O'Shaughnessy: Crisis in long bonds is imminent)
"I can only say: I'm sorry, America," he wrote. "The central bank continues to spin QE as a tool for helping Main Street. But I've come to recognize the program for what it really is: the greatest backdoor Wall Street bailout of all time."
Huszar told "Fast Money" that the bond-buying program, which was supposed to increase credit availability to consumers and businesses, didn't do so.
"There was actually a net decrease in mortgage lending," he said. "In fact, until 2012 mortgage lending was at a 15-year low."
(Read more: Dennis Gartman sees 'a massive top' in bonds)
QE's de facto support of the stock market left out a sizable portion of the people it was supposed to help, Huszar added.
"Let's be honest, 50 percent of Americans don't own stock," he said. "There's a certain amount of trickle-down monetary policy involved here. And there's a real question as to whether that works compared to the huge costs that the program has."
Huszar also said that QE thwarted the idea of reining in banks "too big to fail."
"By virtue of reflating the markets, we've potentially taken the emphasis out of breaking up what is ultimately a banking cartel in the United States," he said, adding that "0.2 percent of banks control 70 percent of assets in this country."
—By CNBC's Bruno J. Navarro. Follow him on Twitter @Bruno_J_Navarro.