Stocks initially held higher throughout Yellen's comments and the Fed statement. Traders attributed the late-session decline in stocks to concerns about Fed credibility and post-settle declines in oil prices to May 19 lows, amid focus on the Brexit vote.
"I do think the Fed changes their tone a little too quickly on data," said Eric Stein, co-director of global fixed income at Eaton Vance Management.
Given the surprising miss on the May jobs report and Brexit, analysts generally did not expect the Fed to raise rates but were watching for clues on the likelihood of a July hike.
"It feels as though 2016 may be like 2015 where you have one and done," said Art Hogan, chief market strategist at Wunderlich Securities. "It feels like they walked back a little bit."
"We are more concerned about Brexit now than we were about the Fed," he said.
Fed Chair Janet Yellen said in a press conference following the statement release that the Brexit vote was one of the factors in Wednesday's decision.
"She can't do anything about Brexit. One way or the other you're left with one of the major risk factors for the market that she has no say over," said Bill Stone, chief investment strategist at PNC Asset Management.
Treasury yields declined after the statement release, with the 2-year yield falling sharply to near 0.67 percent, its lowest since mid-February, and the 10-year yield near 1.58 percent. Earlier in the day, the 2-year yield hit 0.746 percent, its highest since June 10.
The U.S. dollar index was last about a third of a percent lower, with the euro near $1.126 and the yen around 105.9 yen versus the greenback. Sterling was near $1.42.
"You're basically back to status quo of being data dependent and being hyper-focused on every piece of data that comes in," said Aaron Clark, portfolio manager at GW&K Investment Management.
Gold extended gains after the early afternoon settle to briefly top $1,300 an ounce, hitting its highest since early May.
The Dow transports and Russell 2000 gave up much of their earlier gains to close just mildly higher.
Materials led S&P 500 advancers, with utilities the greatest laggard. Financials clung to gains, while consumer discretionary was among the top advancers with the S&P Retail ETF (XRT) up 1.2 percent.
"I think we're still in a market that still sees a lot of uncertainty. (It) would like to see some good news and push to new highs," said Bruce McCain, chief investment strategist at Key Private Bank.
As of the close Wednesday, the major U.S. stock indexes were down more than 1 percent for the week so far.
Concerns about next Thursday's U.K. vote on whether to leave the European Union have weighed on global stocks in the last few days as recent polls generally indicated greater support for leaving.
U.S. crude oil futures settled down 48 cents, or 0.99 percent, at $48.01 a barrel. Earlier, oil came well off session lows to briefly attempt gains after EIA data showed U.S. crude oil stocks declined 933,000 barrels. Late Tuesday, the American Petroleum Institute said crude inventories rose by 1.2 million barrels versus expectations for 2.3 million barrel decline.
In U.S. economic news, industrial production declined 0.4 percent in May and April's figure was revised lower to show a 0.6 percent increase versus the previously reported 0.7 percent. May capacity utilization edged lower from the prior month to 74.9 percent.
The producer price index rose a slightly more-than-expected 0.4 percent in May, following a 0.2 percent rise the prior month, the Labor Department said. In the 12 months through May, PPI fell 0.1 percent after being unchanged in April.
The core PPI that excludes food, energy and trade services fell 0.1 percent last month after rising 0.3 percent in April, for a 0.8 percent rise in the 12 months through May.
The June Empire State Manufacturing Survey rose to 6.0 from a negative 9 print last month. Sub-index components on new orders and shipments also recovered positive territory, while inventories fell further into negative territory and the employment index was zero.
The German 10-year bund yield traded in negative territory, but off record lows touched Tuesday.
In Europe, the German DAX and STOXX Europe 600 were about 1 percent higher.
The Bank of Japan is set to release its statement on monetary policy overnight.
"If there's anybody that has a penchant for surprise it could be (BOJ Governor Haruhiko) Kuroda. There's a little concern there and possibility of a Bank of Japan move tomorrow," said Ben Pace, CIO of HPM Partners.