Read More Federal debt contracts for first time since 2003
"I think the movement down in yields was surprising," said Jim Caron, global portfolio manager at Morgan Stanley Investment Management. Caron said it's possible there was a "blow off" after the 10-year yield hit 2.5 percent. That could suggest that the market is topping out.
"I've always been of the contention that it's going to be really hard for 10-year Treasury yields to be above 2.5 percent unless the data shows it's gotten better," he said.
Traders had been selling Treasurys this week as new supply was set to come onto the market. New debt was issued by the U.K., Japan, and Italy and there were a number of new corporate issues Thursday, including Morgan Stanley, Caterpillar, Devon Energy and Ford.
Read More US retail sales jump in May, brighten growth outlook
"You were back to 2014 levels" before the rally, said Ward McCarthy. "It's one of these situations where the market backed up enough to attract the buy side. I think we'll continue to see a lot of volatility ahead of the FOMC meeting next week." The Fed isn't expected to take action at its rates meeting Tuesday and Wednesday, but it could adjust its economic forecast and rate hike expectations.
Twitter could also be a highlight, after its stock gained 5 percent in the after hours on news that CEO Dick Costolo resigned.
Read More US import prices up as petroleum posts big gains
What to Watch
8:30 a.m.: PPI
10 a.m.: Consumer sentiment
Read More Contrarian: Greece is going to default—here's why