Stocks roared to new all-time highs and bond yields retreated as the Fed defied the market's conventional thinking by keeping its unconventional bond-buying program intact.
Most major Wall Street banks and firms expected the Fed to slightly pare back its $85 billion monthly bond buying program, by $10 billion to $15 billion. But the Fed said it wasn't ready to cut back, citing a tightening in financial conditions that it said could hurt the economy and employment.
The Fed specifically cited rising mortgage rates, and it also blamed Washington, where Congress is heading toward another showdown on the debt ceiling. "Federal fiscal policy continues to be a restraint on growth and a source of downside risk," said Fed Chairman Ben Bernanke.
Treasury yields fell, and the spread between yields on mortgage securities and Treasurys narrowed. The market had been expecting a slow tapering of the Fed program and interest rates have been rising ahead of that, with the 10-year reaching a high of 3 percent in recent sessions. But after the Fed statement, the yield rapidly fell below 2.7 percent.
(Read more: Twitter on the taper)
"I think it's incredibly wimpy," said David Kelly, chief global strategist at J.P. Morgan Funds. "We have the strongest vehicles sales since 2008. The strongest housing sales since 2009. There is a cost to keep this easy money going." Kelly said it was unclear what kind of economy the Fed would need to see before slowing its program, and it may regret that it did not act sooner if it starts to see inflation pick up.
(Read more: The Ultimate Federal Reserve QE Taper Live Blog)Stocks that would benefit from lower interest rates, like home builders and utilities saw particularly strong gains, as the Dow closed up 147 points at 15,676, and the S&P 500 soared 1.2 percent to 1,725. Both indexes were at new highs.
"In every cylce, the Fed is late. The Fed is never a leading indicator. It's always a lagging indicator and it looks like the Fed is going to be a lagging indicator in this cycle too," said Richard Bernstein, CEO of Richard Bernstein Capital Management.
Bernstein said rates should begin to rise if the economic data improves. "Actually, I think what's going to happen is the stock market will grind a little higher. The odds are if the economy can strengthen, rates will go up," he said.
Bernanke also re-emphasized that the asset purchase program is dependent on economic data, so the market will stay hyper-focused on every release. Thursday's data includes weekly jobless claims at 8:30 a.m. ET, as well as existing-home sales, the Philadelphia Fed survey and leading indicators, all at 10 a.m.
During the press conference, the Fed chairman also said there is no fixed calendar schedule guiding the bond program though he had said previously that the slowdown of purchases could begin before year end and the program could end by mid-2014. That led most Wall Street economists to forecast the beginning of the tapering back of purchases on a time line that started in 2013, with many settling on September.
"I think the tightening of financial conditions in mortgage rates and lending, that trumped the Fed keeping the market's trust. The Fed did promise it would taper later this year. People though ti would be September, but it might be October, might be December. I still think they have to do it," said John Canally, investment strategist and economist with LPL Financial.
Bernanke said the Fed will continue to try communicating clearly on the bond buying program and try not to disturb markets. He describedit as "tools that are less familiar, harder to quantify and harder to communicate than the federal Funds rate."
—By CNBC's Patti Domm. Follow here on Twitter @pattidomm