Overall global growth is expected to expand at 3.1 percent in 2016, according to the IMF's "World Economic Outlook" released Tuesday. That's unchanged from its July forecast. But advanced economies, which include the United States, will slow this year to 1.6 percent growth. That compares with 2.1 percent last year and a July IMF forecast of 1.8 percent.
Gold futures for December delivery fell more than 3 percent to settle at $1,269.70 per ounce, posting its worst trading day since 2013.
"The sell-off which we are seeing for gold is mainly due to the reason that some Fed members have created noise again that November meeting could be live when it comes to the interest rate. Although it sounds extremely bizarre because we also have the US election in that particular month, and I do not see the Fed combining the two risky events together," Naeem Aslam, chief market analyst at Think Markets, said in a note to clients.
Investors also watched Deutsche's stock, after it sent markets around the world for a roller-coaster ride last week as worries that it would not be able to pay its massive fine weighed on investor sentiment. On Tuesday, the German banking giant's U.S.-listed shares rose more than 2 percent.
German newspaper Frankfurter Allgemeine Zeitung reported over the weekend that Deutsche's CEO John Cryan would be in Washington DC to negotiate a settlement with the U.S. government over its $14 billion fine.
"It's very important for investors to read between the lines," said Adam Sarhan, CEO at Sarhan Capital. "Market participants have a way of separating facts and noise. Right now, the fact is that investors have lost confidence in Deutsche Bank."
Dow Jones industrial average 2-day chartSource: FactSet
In Fed news, Richmond Fed President Jeffrey Lacker said there was a strong case to raise interest rates significantly and keep inflation under control. "Pre-emptive increases in the federal funds rate are likely to play a critical role in maintaining the stability of inflation," Lacker, a non-voting member of the Fed's policymaking committee, said in a statement.
U.S. Treasury yields rose after Lacker's speech, with the two-year note yield at 0.82 percent and the benchmark 10-year yield at 1.68 percent.
Chicago Fed President Charles Evans is set to speak at 7:40 p.m. ET, on monetary policy and the economy.
"I think the Fed speeches are going to be noise in the background today," said Art Hogan, chief market strategist at Wunderlich Securities. "Right now, we're looking at a data-heavy week, but most of that is due at the end of the week."
Key data reports due this week include the September jobs report, scheduled for release Friday at 8:30 a.m. ET.
"The market is in a wait-and-see mode. We've got earnings season coming and earnings have contracted even with low interest rates," said Sarhan. "The economy is lackluster and you've got currency fluctuations sometimes adversely affecting earnings."
Earnings season is set to kick into full gear next week, when Alcoa reports quarterly results.
Overseas, European equities rose broadly, as the British FTSE 100 advanced 1.3 percent, while the British pound hit its lowest level in more than 30 years. Earlier on Tuesday, The IMF said uncertainty stemming from the so-called Brexit will dampen investor confidence. It sees the U.K. expanding at only 1.8 percent this year, and sees a deepening slowdown to 1.1 percent next year. The U.K. grew at a 2.2 percent pace last year.
"I think a lot of currency investors figures that the UK would be able to negotiate a special deal," said Chris Gaffney, president of EverBank World Markets. "Calling for a March exit really caught people off guard."
The Dow Jones industrial average closed 85.40 points lower, or 0.47 percent, at 18,168.45, with 3M leading decliners and JPMorgan Chase leading advancers.
The S&P 500 fell 10.71 points, or 0.5 percent, to close at 2,150.49, with utilities leading 10 sectors lower and financials the only advancer.
The Nasdaq fell 11.22 points, or 0.21 percent, to end at 5,289.66.
About three stocks declined for every advancer at the New York Stock Exchange, with and exchange volume of 882.64 million and a composite volume of 3.622 billion at the close.
U.S. crude futures settled 0.25 percent higher, or 12 cents, at $48.69 per barrel.
—CNBC's Evelyn Cheng contributed to this report.
On tap this week:
*Planner subject to change.
7:40 p.m.: Chicago Fed President Charles Evans speaks on monetary policy and the economy
Earnings: Yum Brands, Constellation Brands, Monsanto, RPM International
8:15 a.m.: ADP payrolls
8:30 a.m.: Trade deficit
9:30 a.m.: Minneapolis Fed President Neel Kashkari gives welcoming remarks at an event on "Early Childhood Development in Indian Country"
9:45 a.m.: Markit services PMI
10 a.m.: ISM non-manufacturing
10 a.m.: Factory orders
1 p.m. & 5 p.m.: Richmond Fed President Jeffrey Lacker speaks on "Does Federal Reserve Governance Need Reform?"
Earnings: Helen of Troy, International Speedway, Ruby Tuesday
8:30 a.m.: Jobless claims
2 p.m.: Atlanta Fed live webcast on search process for new bank president and answer questions from the public
8:30 a.m.: Employment report
10 a.m.: Wholesale trade
10:30 a.m.: Federal Reserve Vice Chairman Stanley Fischer speaks on the economy and financial regulation
12:45 p.m.: Cleveland Fed President Loretta Mester speaks on Fed communications
3 p.m.: Consumer credit
3:40 p.m.: San Francisco Senior Vice President Mary Daly speaks on the U.S. economic outlook and the role of education in supporting the American dream
4 p.m.: Federal Reserve Governor Lael Brainard speaks on blockchain technology
*All times Eastern. Planner subject to change.