"There has been a lot of chatter about the Fed being behind the curve. I think it's going to be important to see what we get ... in the inflation numbers and wage numbers" this week, said Michael Feroli, chief economist at JPMorgan.
The July employment report and personal income and spending data is reported Friday. ADP's private sector employment report Wednesday showed a gain of 218,000 jobs.
Stocks rose and bond yields climbed, but stocks later slumped while bond yields held higher levels. The 10-year yield was at 2.52 percent in late morning trading, well above the 2.46 percent level just before the release of GDP.
Yields were higher across the curve, with the 3-year above 1 percent, a level it has not consistently held since 2011. The 2-year yield reached 0.579 percent, the highest since May 2011. The short end of the curve is where much of the rate hike speculation has showed up.
The improvements to the economy in the GDP report were broad, with strengthening in government, business and the consumer. Consumer spending added 2.5 percent, better than the 1.9 percent expected.
"This market wants to read something bearish into it," said David Ader, chief Treasury strategist at CRT Capital. Bad news, or bearish news for bonds is often good news for the economy.
Feroli said the pace of growth in the second half could be about 2.5 percent, and that there should be some giveback from the second quarter's build in inventories.
Read MoreWhy the Fed could raise rates sooner than you think
The number also shows a big bounce from the revised negative 2.1 percent first quarter. The consensus among enomicsts for Q3 growth remained at 3.2 percent Wednesday, according to a CNBC/Moody's rapid update.
"That big contribution from inventories being accumulated at over a $90 billion pace in the second quarter will weigh on, I think somewhat on activity in the third quarter," Feroli said on CNBC's "Squawk Box." While looking back I think this is a better-than- expected picture for the first half as a whole, looking forward it may be a bit of a headwind in terms of businesses having to destock after having accumulated so many inventories in the second quarter."
The last time GDP was above a 4 percent growth rate was third quarter of last year, when it was 4.5 percent. GDP for 2013 was revised higher to 3.1 percent, from 2.6 percent.
"The Fed has to be delighted with this. This also explains there had been a significant disconnect between an obviously improving labor market and growth measures, which had remained frustratingly low," said Ward McCarthy, chief financial economist at Jefferies. "I think what we're finding is the disconnect wasn't really as acute as prior data had made it appear,"
The Fed is watching both inflation and the labor market as it weighs when it will further reduce stimulus by cutting rates. It is expected to finish tapering its bond buying program in October, and the market has been full of talk the Fed will move to cut rates sooner than the mid-2015 timetable that many economists expect.
But McCarthy said the Fed will not be swayed by the improving picture. "They're looking for sustained growth, and this follows a pretty dismal first quarter," he said. "This makes me much more confident the last two quarters of 2014 growth is going to average 3 percent or more."
Joseph LaVorgna, Deutsche Bank's chief U.S. economist, was the highest on the Street with his forecast of 4.2 percent second quarter growth, and he sees a strong 3.5 percent pace in the second half.
Read MoreIt does not feel like a 4% economy: Aetna CEO
He said the pickup in inflation is not surprising, but the Fed is not likely to move any faster to hike rates.
"Tightening sooner means they have to tighten less," he said.
—By CNBC's Patti Domm