Six of the 10 S&P 500 sectors were in negative territory on Thursday. The S&P financial index slipped 0.3 percent and ranked among those leading the market's decline.
St. Louis Fed President James Bullard, in an interview with Fox Business Network, reiterated his belief that raising rates by the end of the first quarter of 2015 would be appropriate.
He said the U.S. jobless rate will fall below 6 percent and inflation looks likely to rise back to 2 percent later this year, putting the economy closer to normal than most realize. Bullard is a non-voting member of the Federal Open Markets Committee, the Fed's policy-making panel.
The Fed had hinted after its meeting this month about a slightly faster pace of interest-rate increases starting next year, but suggested rates in the long run would be lower than it had indicated previously.
Read MoreJim Cramer: The best bets for the second half
U.S.-listed shares of Barclays dropped 7.4 percent to $14.55 after New York State's attorney general filed a securities fraud lawsuit against Barclays, accusing the British bank of giving an unfair edge in the United States to high-frequency trading clients.
The Barclays fallout hit other European banks. U.S.-listed shares of UBS slid 2.4 percent to $18.47. Credit Suisse fell 3.6 percent to $28.29.
The Dow Jones industrial average fell 21.38 points or 0.13 percent, to end at 16,846.13. The S&P 500 slipped 2.31 points or 0.12 percent, to 1,957.22. The Nasdaq Composite dipped 0.71 of a point or 0.02 percent, to 4,379.05.
The CBOE Volatility Index, Wall Street's fear barometer, edged up just 0.4 percent to 11.63, or about half of its historical average. That makes sense, given the stock market's recent record highs, analysts said.
"As we go higher, volatility should be decreasing," said JJ Kinahan, chief derivatives officer at TD Ameritrade.