After all, the Fed currently buys $85 billion a month in U.S. Treasurys and mortgage-backed securities, on pace to total roughly $1 trillion a year. To begin tapering its bond-buying program, Gross said the Fed will need the private market to assume those purchases.
(CNBC Explains: What is the Federal Reserve?)
Why would the private market buy what the Fed is buying now? It'd only happen under two conditions, Gross said.
"If yields were sufficiently high and perhaps they've done that over, you know, the past several quarters," Gross said. "But as well, if the funding of those positions was guaranteed in terms of its cost and so that's where [Fed chair nominee] Janet Yellen comes in. That's where the perfect, you know, path comes in if, in fact, it can be accomplished."
(Read more: What to expect in Yellen hearing: El-Erian)
Yellen, President Barack Obama's pick to succeed Fed Chairman Ben Bernanke, must convince investors that the cost of leverage will remain at 25 basis points for the next several years, Gross said. Even then, the manager of the world's largest bond fund is doubtful the transition will be without economic consequence.
(CNBC Explains: What is quantitative easing?)
"It's fair to say that quantitative easing and, you know, five years of 25-basis-point policy rates have not really generated much in the way of economic growth," Gross said, adding that a paltry 2 percent gross domestic product is now the "new normal."
"If that's the best they can do," he said, "what will be done going forward if tapering and quantitative easing pulls back?"
Yellen is scheduled to appear before Congress for her confirmation hearings on Thursday.