U.S. crude oil futures came well off session lows to settle 0.8 percent lower at $39.46 a barrel. The weekly oil rig count, released Thursday due to the Good Friday holiday, showed a drop of 15 oil rigs.
The U.S. dollar index pared gains after touching its highest in more than a week, but turned in its first positive week in four with a rise of more than 1 percent. The euro held near $1.118 and the yen was at 112.81 yen against the greenback.
Gold futures for April delivery settled 0.2 percent lower at $1,221.60 an ounce, down 2.61 percent for the week, its worst since Nov. 6.
Treasury yields held higher, with the 2-year yield near 0.88 percent and the 10-year yield around 1.9 percent. The bond market closed early Thursday, at 2 p.m. ET.
"The real reason for the market coming down in the last couple of days is the Fed rhetoric," said Peter Cardillo, chief market economist at First Standard Financial.
"Certainly the comments out of Bullard and some of the others indicates we're getting closer to a rate hike," he said.
Ahead of the open, St. Louis Federal Reserve President James Bullard said the Fed's next interest rate hike "may not be far off," provided the economy evolves as expected. However, he told reporters in a Reuters article he was undecided on whether to push for a rate hike at the Fed's April meeting partly because of the limited amount of data expected before then.
Bullard is a voting member of the Federal Open Market Committee and in an interview Wednesday was the latest Fed official to suggest rates could rise as early as April.
Read MoreTraders look to data as Fed confuses
"I think it's adding to the volatility, especially with the results of the meeting more dovish than expected, countered by commentary more similar to what we expect," said Kate Warne, investment strategist at Edward Jones.
Last week, the U.S. central bank left rates unchanged and lowered its projections for the number of hikes this year.
The Dow transports ended nearly 0.4 percent lower, well off session lows but still 1.85 percent lower for the week to end nine straight weeks of gains. American Airlines was the greatest decliner for the day and week.
"I think right now we're in pullback mode. There's a lot of profit-taking, end-of-month and end-of-quarter," Sarhan said of the morning decline in stocks. He's watching to see if buyers come in to support the market, or if the selling continues.
Read More Market calm has eerie parallels to pre-swoon August
"What's most important in my eyes is people are looking forward to employment next week," said JJ Kinahan, chief strategist at TD Ameritrade.
"As long as we hold above 2,000 (in the S&P) there's no reason to panic," he said.
In economic news, weekly jobless claims came in at 265,000. February durable goods orders declined 2.8 percent.
Non-defense capital goods orders excluding aircraft, a closely watched proxy for business spending plans, decreased 1.8 percent after advancing by a downwardly revised 3.1 percent in January, Reuters said.
"The durable goods report was clearly disappointing," Warne said. "I don't think anybody was surprised."
The Markit flash U.S. services PMI was 51.0 in March, up from 49.7 in February.