The Dow and S&P closed little changed Wednesday after the Federal Open Market Committee's April meeting minutes released that afternoon said a June rate hike was likely if the data improves.
"As far as the bond market goes, I still think interest rates are going to stay low. I'm not a big believer in what (the Fed policymakers have) been saying," said John Caruso, senior market strategist at RJO Futures.
"Anytime you have talk of higher interest rates stocks throw a temper tantrum," he said.
Treasury yields held lower, with the 10-year yield near 1.85 percent. The 2-year yield was around 0.88 percent, coming off highs not seen since mid-March that it touched Wednesday.
Financials and industrials were the worst S&P performers on the day, while utilities led advancers. JJ Kinahan, chief strategist at TD Ameritrade, said financials were reacting to the reversal in bond yields, while utilities bounced after falling Wednesday.
"I think what it shows is a lack of conviction everywhere," he said.
Market expectations for a June rate hike moderated to a 28 percent chance Thursday afternoon, after rising to 34 percent Wednesday afternoon, according to CME Group's FedWatch tool. The probability for a July hike was 52 percent, a touch above Wednesday's 51 percent chance.
Shares of Wal-Mart rose after better-than-expected quarterly results. The retailer also reported a 1 percent increase in U.S. same-store sales, topping estimates of a half percent increase.
The SPDR S&P Retail ETF (XRT) closed nearly 1.3 percent higher, but still about 12 percent lower for the quarter so far.
"I think (Wal-Mart) is a big factor helping the market today. The market's probably better than it would have been in light of higher interest rates," said Jack Ablin, chief investment officer at BMO Private Bank.
Read MoreWhy London could decide the next US rate hike
"In Japan and Europe to some degree (negative interest rate policy has) backfired. I think what it's really doing is it's more harmful to the banking system than the benefits ... to the economy," he said. "Here it's clearly the prospect of higher rates. It's really the other side of the same coin. When real rates rise currencies rise and stimulus is pared."
The U.S. dollar index hit highs not seen since late March before trimming gains to hold about 0.25 percent higher. traded about 0.2 percent higher after earlier hitting levels not seen since late March. The euro was near $1.120 and the yen around 109.9 yen against the greenback as of 3:22 p.m. ET.
The iShares MSCI Emerging Markets ETF (EEM) closed 0.9 percent lower, in the red for the year so far.
As of the close Thursday, the major averages were tracking for a weekly decline. The Dow and S&P were on pace for their first four-week losing streak since the one ended October 2014, while the Nasdaq was on pace for its first five-week losing streak since 2012.
"The SPX has confirmed a breakdown below its 50-day moving average, along with about half of its constituents in a loss of market breadth," BTIG Chief Technical Strategist Katie Stockton said in a note.
"Short-term momentum is weak, but oversold conditions could help minimize immediate downside follow-through," she said. "We would be wary of a breakdown below intraday support (near 2039 for the SPX), noting the next point of reference on the chart is at the 200-day moving average (near 2012)."
Read MoreCheapest Memorial Day gas in 11 years
In the day's economic news, weekly jobless claims declined to 278,000, after three weeks of increases. The May Philly Fed index showed minus 1.8.
Also weighing on stocks were morning comments by Richmond Fed President Jeffrey Lacker on Bloomberg Radio that said he was comfortable with four Fed rate hikes in 2016.
According to prepared remarks Thursday morning, Fed Vice Chair Stanley Fischer said in order to lift the long-run equilibrium interest rate what the United States needs most is "faster potential growth" now that the economy is near full employment and approaching the Fed's target inflation rate.
He did not comment on policy or the economic outlook.
European stocks closed lower, with the German DAX off nearly 1.5 percent. Asian stocks ended mostly lower, with the Hang Seng down nearly 0.7 percent and the Nikkei 225 0.01 percent higher.
Read MoreEarly movers: WMT, DKS, AAP, TSLA, MON, FMC, CHD, CRM, URBN & more
The Dow Jones industrial average closed down 91.22 points, or 0.52 percent, at 17,435.40, with Goldman Sachs leading decliners and Wal-Mart the top gainer.
The S&P 500 closed down 7.59 points, or 0.37 percent, at 2,040.04, with industrials leading six sectors lower and utilities the top gainer.
The Nasdaq composite closed down 26.59 points, or 0.56 percent, at 4,712.53.
The CBOE Volatility Index (VIX), widely considered the best gauge of fear in the market, traded higher around 16.4 after earlier hitting 17.65, its highest since March 15.
About three stocks declined for every advancer on the New York Stock Exchange, with an exchange volume of 934 million and a composite volume of nearly 3.8 billion in the close.
Gold futures for June delivery settled down $19.60 at $1,254.80 an ounce.
—Reuters contributed to this report.
On tap this week:
Earnings: Campbell Soup, Deere, Foot Locker, The Buckle
9 a.m.: Fed Governor Daniel Tarullo on insurance company supervision and regulation
10 a.m. Existing home sales
*Planner subject to change.
More from CNBC.com: