U.S. stocks closed lower Thursday after comments from New York Fed President William Dudley and the latest Fed minutes increased prospects of a rate hike as soon as June.
The S&P 500 erased gains for the year so far (Tweet This), while the Dow Jones industrial average clung to gains of 0.06 percent for the year so far. Both indexes were about 5 percent or less below their 52-week intraday highs touched last May 20 and 19, respectively.
"It's really repricing our estimates of when the Fed hikes next," said Art Hogan, chief market strategist at Wunderlich Securities, noting consensus has shifted from later in the year to July.
The major averages came well off session lows in afternoon trade, with the Dow ending about 91 points lower after earlier falling 195 points. Wal-Mart closed 9.5 percent higher to contribute the most to gains, while Goldman Sachs had the greatest negative impact, giving up most of its gains from Wednesday.
"I think it's just hangover from yesterday," Jeremy Klein, chief market strategist at FBN Securities, said, noting stocks came off lows as oil pared losses, and that Dudley's remarks were more a reminder the Fed could still move this summer.
Stocks are "kind of hanging in there, which is encouraging," he said.
Traders also attributed part of the late-session recovery in stocks to options expiration Friday.
U.S. crude oil futures for June delivery settled down 3 cents, or 0.06 percent, at $48.16 a barrel, well off session lows amid more concerns of near-term supply outages from Nigeria.
The major U.S. stock indexes fell to session lows Thursday morning amid Dudley's comments that June is definitely a live meeting and he was quite pleased market expectations for the probability of a June or July rate hike has moved up.
"The comment that the market was right to readjust isn't saying it will raise rates," said Peter Boockvar, chief market analyst at The Lindsey Group, pointing out that the S&P fell but the 2-year yield was not moving.
Dudley also said the "Brexit" vote on whether to Britain should remain in the European Union complicated the Fed's decision to raise rates.
The Dow and S&P closed little changed Wednesday after the Federal Open Market Committee's April meeting minutes released that afternoon said a June rate hike was likely if the data improves.
"As far as the bond market goes, I still think interest rates are going to stay low. I'm not a big believer in what (the Fed policymakers have) been saying," said John Caruso, senior market strategist at RJO Futures.
"Anytime you have talk of higher interest rates stocks throw a temper tantrum," he said.
Treasury yields held lower, with the 10-year yield near 1.85 percent. The was around 0.88 percent, coming off highs not seen since mid-March that it touched Wednesday.
Financials and industrials were the worst S&P performers on the day, while utilities led advancers. JJ Kinahan, chief strategist at TD Ameritrade, said financials were reacting to the reversal in bond yields, while utilities bounced after falling Wednesday.
"I think what it shows is a lack of conviction everywhere," he said.
Market expectations for a June rate hike moderated to a 28 percent chance Thursday afternoon, after rising to 34 percent Wednesday afternoon, according to CME Group's FedWatch tool. The probability for a July hike was 52 percent, a touch above Wednesday's 51 percent chance.
Shares of Wal-Mart rose after better-than-expected quarterly results. The retailer also reported a 1 percent increase in U.S. same-store sales, topping estimates of a half percent increase.
The SPDR S&P Retail ETF (XRT) closed nearly 1.3 percent higher, but still about 12 percent lower for the quarter so far.
"I think (Wal-Mart) is a big factor helping the market today. The market's probably better than it would have been in light of higher interest rates," said Jack Ablin, chief investment officer at BMO Private Bank.
Read MoreWhy London could decide the next US rate hike
"In Japan and Europe to some degree (negative interest rate policy has) backfired. I think what it's really doing is it's more harmful to the banking system than the benefits ... to the economy," he said. "Here it's clearly the prospect of higher rates. It's really the other side of the same coin. When real rates rise currencies rise and stimulus is pared."
The U.S. dollar index hit highs not seen since late March before trimming gains to hold about 0.25 percent higher. traded about 0.2 percent higher after earlier hitting levels not seen since late March. The euro was near $1.120 and the yen around 109.9 yen against the greenback as of 3:22 p.m. ET.
The iShares MSCI Emerging Markets ETF (EEM) closed 0.9 percent lower, in the red for the year so far.
As of the close Thursday, the major averages were tracking for a weekly decline. The Dow and S&P were on pace for their first four-week losing streak since the one ended October 2014, while the Nasdaq was on pace for its first five-week losing streak since 2012.
"The SPX has confirmed a breakdown below its 50-day moving average, along with about half of its constituents in a loss of market breadth," BTIG Chief Technical Strategist Katie Stockton said in a note.
"Short-term momentum is weak, but oversold conditions could help minimize immediate downside follow-through," she said. "We would be wary of a breakdown below intraday support (near 2039 for the SPX), noting the next point of reference on the chart is at the 200-day moving average (near 2012)."
Read MoreCheapest Memorial Day gas in 11 years
In the day's economic news, weekly jobless claims declined to 278,000, after three weeks of increases. The May Philly Fed index showed minus 1.8.
Also weighing on stocks were morning comments by Richmond Fed President Jeffrey Lacker on Bloomberg Radio that said he was comfortable with four Fed rate hikes in 2016.
According to prepared remarks Thursday morning, Fed Vice Chair Stanley Fischer said in order to lift the long-run equilibrium interest rate what the United States needs most is "faster potential growth" now that the economy is near full employment and approaching the Fed's target inflation rate.
He did not comment on policy or the economic outlook.
Major U.S. Indexes
European stocks closed lower, with the German DAX off nearly 1.5 percent. Asian stocks ended mostly lower, with the Hang Seng down nearly 0.7 percent and the Nikkei 225 0.01 percent higher.
Read MoreEarly movers: WMT, DKS, AAP, TSLA, MON, FMC, CHD, CRM, URBN & more
The Dow Jones industrial average closed down 91.22 points, or 0.52 percent, at 17,435.40, with Goldman Sachs leading decliners and Wal-Mart the top gainer.
The closed down 7.59 points, or 0.37 percent, at 2,040.04, with industrials leading six sectors lower and utilities the top gainer.
The Nasdaq composite closed down 26.59 points, or 0.56 percent, at 4,712.53.
The CBOE Volatility Index (VIX), widely considered the best gauge of fear in the market, traded higher around 16.4 after earlier hitting 17.65, its highest since March 15.
About three stocks declined for every advancer on the New York Stock Exchange, with an exchange volume of 934 million and a composite volume of nearly 3.8 billion in the close.
Gold futures for June delivery settled down $19.60 at $1,254.80 an ounce.
—Reuters contributed to this report.
On tap this week:
Friday
Earnings: Campbell Soup, Deere, Foot Locker, The Buckle
9 a.m.: Fed Governor Daniel Tarullo on insurance company supervision and regulation
10 a.m. Existing home sales
*Planner subject to change.
More from CNBC.com: