Stocks recovered from their session lows but still finished in the red, with the Dow snapping its Tuesday winning streak, amid ongoing worries over the future of the Fed's bond-buying policy and ahead of Japanese Prime Minister Shinzo Abe's speech.
"Yesterday was the first day that the Pavlovian interconnection between the after-markets in Japan and here got a bit more muted…and we're seeing a muted influence again today," said Art Cashin, director of floor operations at UBS Financial Services. "[Japan's Prime Minister] Mr. Abe tomorrow is due to drop a couple of shoes and people are waiting to see what that may bring so there's slight anxiety."
The Dow Jones Industrial Average slumped 76.49 points, to end at 15,177.54, dragged by Home Depot and Microsoft, breaking its streak of 20-consecutive Tuesdays in positive territory. The Dow was down nearly 150 points at its lows.
The CBOE Volatility Index (VIX), widely considered the best gauge of fear in the market, ended above 16.
Most key S&P sectors ended lower, led by financials and energy, while telecoms held modest gains.
"We've started to get a little choppy since the end of May," noted John Fox, co-manager of the FAM Value Fund. "There's change in the wind and it's not the same market as before: Volatility's increased in the last couple of days, economic data's been a mixed bag and the momentum in the market's changed."
Intraday swings have increased in last few weeks as investors continue to question when the Fed may curtail its bond-buying program.
Most recently, Kansas City Fed President Esther George, reiterated her support for tapering the bond-buying program and said slowing down QE3 would help wean financial markets off their dependence on ultra-easy money from the central bank.
Investors will be monitoring Japan's Prime Minister Shinzo Abe' sspeech on growth strategy on Wednesday. Separately, a report from the Nikkei that there is growing speculation that the the Bank of Japan will soon be left with no room to purchase REITs. The report also said many investors are hoping the BoJ raises the ceiling at its two-day policy meeting starting Monday.
In addition, the monthly government employment data on Friday will be closely watched as the report is a key factor for the Fed's decision on monetary policy. Economists surveyed by Reuters expect to see a gain of 170,000 jobs, slightly higher than the 165,000 jobs added in April.
Stocks in Asia closed mixed, with Japanese equities still volatile after the dollar-yen fell below the key 100-mark. However, European markets were higher, boosted by data showing a decline in unemployment numbers in Spain.
(Read More: Traders Debate Fed Moves While Japan in Focus)
Zynga edged higher a day after the social videogame maker said it plans to cut 520 jobs, or approximately one-fifth of its workforce, in an effort to reduce costs and restructure its business. Shares initially fell as much as 14 percent following the announcement Monday afternoon.
ExactTarget skyrocketed more than 50 percent after Salesforce.com said it will acquire the marketing software company for nearly $2.5 billion. Many high-profile companies including Coca-Cola, Gap and Nike use ExactTarget to manage their digital marketing.
Dollar General slumped after the discount chain slashed the top end of its full-year earnings outlook, due to moderating sales growth and a lower gross profit rate.
(Read More: Here Come the 2Q Earnings Warnings)
On the economic front, the U.S. trade deficit widened less than expected in April to $40.3 billion, according to the Commerce Department. Economists polled by Reuters had expected the trade deficit to rise to $41.0 billion in April.
—By CNBC's JeeYeon Park. Follow JeeYeon on Twitter: @JeeYeonParkCNBC
What's Happening This Week:
WEDNESDAY: MBA mortgage applications, ADP employment report, productivity & costs, factory orders, ISM non-mfg index, oil inventories, Beige Book, Las Vegas Sands shareholder mtg, Under Armour investor day, Yelp shareholder mtg; Earnings from Hovnanian, VeriFone
THURSDAY: Bank of England announcement, Challenger job-cut report, ECB announcement, jobless claims, quarterly services survey, natural gas inventories, Fed balance sheet/money supply, chain store sales, Wal-Mart shareholder mtg, Google shareholder mtg, GM annual mtg; Earnings from Ann, JM Smucker, Cooper Cos., Vail Resorts
FRIDAY: Employment situation, consumer credit
More From CNBC.com: