The Dow Jones Industrial Average declined 40.27 points, or 0.30 percent, to end at 13,553.10, led by BofA and Alcoa .
The S&P 500 erased 4.58 points, or 0.31 percent, to close at 1,461.19. The Nasdaq slipped 5.28 points, or 0.17 percent, to finish at 3,178.67.
The CBOE Volatility Index, widely considered the best gauge of fear in the market, gained near 15.
Despite the day's pullback, all three major averages are still up more than 3 percent for the month.
Among the key S&P sectors, materials led the laggards, while health care ended higher.
Stocks took another leg lower around 2pm ET as oil briefly plunged $4 to below $95 a barrel. Some traders passed around rumors about a "fat-finger" trading error, while others buzzed about a potential release from the Strategic Petroleum Reserve.
But the CME Group said it is "unaware of any technical issues" regarding the crude price drop and the White House said there are no immediate plans to make changes to the SPR.
“We’ve had back-to-back terrific weeks with the markets reacting to positive news positively and ignoring negative news—so we’re taking a breather,” said Art Hogan, managing director of Lazard Capital Markets. “Still, this is not the beginning of a pullback—the Bernanke put is going to be here for a while, which will limit some downside risk.”
Stocks logged a four-day rally last Friday and posted sharp gains for the second-straight week as investors cheered the Federal Reserve’s decision to embark on another round of monetary stimulus. (Read More: How Fed's 'Shock and Awe' Will Affect Investors—O'Neill)
“We’re likely to see a 7 to 8 percent pullback in the next six weeks,” said Yu-Dee Chang, chief trader at ACE Investment Strategists. “There are still a lot of uncertainties ahead, but by the middle of November, I see another wave up.”