U.S. stocks closed lower Wednesday, after a solid start to November, weighed by a decline in energy stocks and increased confidence in the possibility of a December rate hike. (Tweet This)
The major averages are still up about 1 percent or more week-to-date, with energy the best performing S&P sector for the week so far.
"The market was overbought in the short term and it was looking for a reason to pause and we got the reason today — oil giving back some of its gains and Yellen upping the (probability) of a rate hike but not committing to it," said Peter Cardillo, chief market economist at Rockwell Global Capital.
Stocks attempted to recover opening gains in intraday trade but still closed lower. UnitedHealth and Walt Disney were the greatest weights on the Dow Jones industrial average.
Energy ended a five-day win streak to fall about 1 percent as the greatest decliner in the S&P 500, which briefly dropped below the psychologically key 2,100 level in intraday trade.
Oil turned lower and extended losses after weekly crude inventories rose for a sixth straight week. Crude settled down $1.58, or 3.3 percent, at $46.32 a barrel.
The Nasdaq composite closed a touch lower. The iShares Nasdaq Biotechnology ETF (IBB) closed down about 0.4 percent, while Apple ended about half a percent lower and Microsoft up about 0.46 percent. The Nasdaq 100 ended a touch lower after a record close Tuesday.
In prepared remarks for her testimony before the House Financial Services Committee on bank regulation and supervision, she said banks are much healthier but problems remain.
Treasury yields held higher after Yellen's mid-morning comments, with the 10-year at 2.23 percent and the at 0.82 percent, its highest level since April 2011.
The U.S. Treasury Department auctioned $26 billion in 2-year notes at a high yield of 0.824 percent.
"Treasury yields (are) starting to push up a little higher as people seem to price in the probability the Fed is going to do something in December," said Jeffery Elswick, director of fixed income at Frost Investment Advisors.
Still, "we're starting to get to this point where some of this momentum from October has started to slow down. ... We're going to need a new catalyst" as we get closer to the Fed's December meeting, he said.
The U.S. dollar traded about 0.8 percent higher against major world currencies, with the euro near $1.08, touching its lowest level since August. The yen traded near 121.57 yen against the greenback.
In the afternoon, New York Fed President William that December is still a possibility, Reuters said. His speech on looking beyond the macro economy did not discuss the current state of U.S. monetary policy.
Fed Vice Chair Stanley Fischer speaks at 7:30 p.m. on central bank independence at the National Economists Club.
"This is the crescendo of Fed speakers. You have Dudley, Fischer and Yellen, and I think all of that helps because it coalesces around (raising rates in) December," said Art Hogan, chief market strategist at Wunderlich Securities.
Before the open, Fed Gov. Lael Brainard spoke in Frankfurt, saying the Fed is "very dependent" on incoming data, but that there have been some tightening of economic conditions.
Philadelphia Fed President Patrick Harker did not comment on the U.S. economy and monetary policy in prepared remarks for a speech on social innovation capital, Reuters reported.
Read MoreTraders brace for Fed, data deluge
Economic data gave a somewhat mixed view on the services sector, which remains well in expansion territory. The October report showed a rise to 59.1 from 56.9 the prior month. However, Markit said its October PMI services index declined to 54.8 from 55.1 in September.
With two days to Friday's nonfarm payrolls report, the ADP employment report showed private companies added 182,000 jobs in October.
came in at $40.8 billion, its lowest level in seven months, Reuters reported.
In other economic news, weekly mortgage applications fell 0.8 percent last week as interest rates rose.
Stock index futures held higher after the data releases, with the Dow futures up about 40 points.
European stocks closed mixed, with Volkswagen weighing on the DAX but the STOXX Europe 600 up about half a percent.
Glencore gained more than 5 percent in London trade after the firm said it was on track to reduce its debt thanks to asset sales and was making deeper cuts in copper output to support weak prices, Reuters reported.
In earnings news, Tesla reported an adjusted quarterly loss and in-line revenue, but focused more on an upbeat production outlook for the electric car maker.
"I do think the market has digested the impact of the dollar, slowdown in China, (and low) oil prices into earnings," said Maris Ogg, president of Tower Bridge Advisors. "Now earnings can stand on their own again and we have much easier comparisons. People are recognizing that next year is not going to be a bad year."
"I think the market has a much healthier outlook. ... People understand the economy is not weakening. We don't have a recession staring at us," she said.
The closed down 7.48 points, or 0.35 percent, at 2,102.31, with energy leading eight sectors lower and utilities and information technology the only advancers.
The Nasdaq closed down 2.65 points, or 0.05 percent, at 5,142.48.
The CBOE Volatility Index (VIX), widely considered the best gauge of fear in the market, held near 15.
About three stocks declined for every two advancers on the New York Stock Exchange, with an exchange volume of 933 million and a composite volume of nearly 4.1 billion in the close.
Gold futures settled down $7.90 at $1,106.20 an ounce.
—Reuters and CNBC's Peter Schacknow contributed to this report.
On tap this week:
Earnings: Facebook, MetLife, Qualcomm, CF Industries, Whole Foods, GoDaddy, King Digital
7:30 p.m.: Fed Vice Chair Stanley Fischer on central bank independence before National Economists Club
Earnings: AstraZeneca, Toyota Motors, Molson Coors Brewing, Disney,Kraft Heinz, Monster Beverage, News Corp., Nvidia, Symantec, TripAdvisor, Dreamworks Animation, Shake Shack
Chicago Fed President Charles Evans gives welcoming remarks at banking conference
8:30 a.m.: Jobless claims, productivity and costs
8:30 a.m.: Philadelphia Fed's Patrick Harker on energy interdependence
8:30 a.m.: New York Fed's Bill Dudley opening remarks at financial services industry conference
9:10 a.m.: Fed Vice Chair Stanley Fischer on IMF panel on reforming financial services
1:25 p.m.: Fed Gov. Dan Tarullo on regulation of international banks at Chicago bank conference
1:30 p.m.: Atlanta Fed President Dennis Lockhart at Joint Central Bank conference, Switzerland
4 p.m.: Former Fed Chairman Ben Bernanke on Fed policy at IMF conference
Earnings: Allianz, Cigna, Humana
8:30 a.m.: October employment report
9:10 a.m.: St. Louis Fed President James Bullard on policy and economy
3:00 p.m.: Consumer credit
4:15 p.m.: Fed Gov. Lael Brainard on policy panel at IMFconference
11:10 p.m.: San Francisco Fed President John Williams on outlook and economic education
More From CNBC.com: