Swonk said while the quarter-over-quarter inflation number packed a surprise, the less volatile year-over-year quarterly core PCE watched by the Fed remains low. "The year-over-year is still tracking at 1.3, and that's basically what it has been. From the Fed's perspective, they expected a bigger impact on core from falling energy prices and lower import prices than occurred," she said.
Read MoreGDP gains speed on consumer spending
Swonk said the Fed will also need the employment cost index, released Friday morning, to again show improvement after a surprise 0.7 percent increase last quarter. There is also important monthly PCE, personal consumption expenditure data released Monday.
While short end yields moved higher Thursday, yields at the long end of the Treasury curve moved lower. The 10-year yield fell to 2.27 percent.
CRT Capital chief Treasury strategist David Ader said the long end was also focused on the soft GDP report, though with revisions, the first quarter showed positive growth of 0.6 percent, from a 0.2 percent contraction.
"The economy's not fantastic, and we have a need for duration. You put a Fed hike in here, and you put in a month-end bid, and you've got a curve flattening going on here," he said.
Read MoreGovernment: years of growth weaker than we thought
Ader said that while there was an uptick in inflation, it could be passing. "Bearing in mind, that oil is now down 20 percent from the average rate of the last quarter, so you're going to see something potentially coming into play, where there will be reduced stressed and the dollar has improved a little bit, so going forward there are some offsetting elements to that," he said.
The moves in the bond market and the picking apart of the GDP report highlights the dichotomy in the market over whether the Fed will raise rates or not, with a slow-growing economy and few signs of inflation.
But the Fed has made it clear it wants to hike rates this year, and some in the market fear that it must raise rates to avoid unintended consequences of its zero-rate policy.
Read MoreNot yet: Fed keeps rates at zero
Dan Greenhaus, chief global strategist at BTIG, said he expects the Fed to raise rates in September regardless of the slow growth. "I think there's a little element of overthinking here," he said.
Greenhaus said the Fed statement Wednesday was much more important when looking at its intentions.
In that statement, the Fed upgraded its view of the economy and labor, but left questions about the course of inflation, which is tracking weaker than the Fed would like. It did say it is "reasonably confident" inflation will move back to its 2 percent objective over the medium term.
The central bank noted that the economy is expanding moderately, dropping a reference to its being little changed in the first quarter. "The labor market continued to improve, with solid job gains and declining unemployment. On balance, a range of labor market indicators suggests that underutilization of labor resources has diminished since early this year," the Fed said.
Read MoreFed tea leaves have market wanting more
The Fed also added the word "some" in a sentence where it described the further improvement it would like to see in labor. Strategists read the new "some further improvement" as a more hawkish tempering of language, suggesting it sees the labor market improvement as very nearly sufficient.