"It's not only a zero percent funds rate. If you assume that the Fed has $4.5 trillion on the balance sheet you have an effective funds rate well below zero percent," the CIO of fundamental fixed income told CNBC's "Squawk Box" in an interview.
"That's not the right level, given what growth is in the economy and given where we're going," said Rieder, who oversees more than $720 billion in fixed income assets at BlackRock and is called by some the new bond king. He sees any rate hike as gradual. "It's not that scary."
He said he does not see bubble conditions in credit markets. "[But] if you don't let things normalize, you can get there," he continued. "If you distort asset prices for too long ... you are starting to see leverage build."
Investors continue to wonder whether the Fed will increase rates for the first time since 2006 next week or wait until December or even next year.
The government's August employment report, released Friday morning, did not seem to settle the rate issue either way.
The unemployment rate was rosier than forecast—falling to 5.1 percent, the lowest since April 2008. But nonfarm job growth of 173,000 came in lower than expectations of 200,000.
The continued financial market volatility on concerns about China's economy has also been seen as a complicating factor for the Fed.
Central bankers have said they would like to hike rates sometime this year should U.S. economic numbers continue to strengthen.