"The main driver behind the down move is that more market participants now believe that the Fed will begin raising rates this year," Commerzbank analyst Daniel Briesemann said.
Wall Street stocks, unable to follow a rally in European and Asian stocks, fell further after Yellen's comments. The U.S. dollar jumped 0.9 percent against a basket of major currencies, building on a rise that followed earlier data showing stronger-than-expected private-sector U.S. job growth.
Oil and other commodity markets tumbled, with the 19-market Thomson Reuters CoreCommodity Index falling 1.7 percent.
"Quite often whenever the U.S. dollar rallies and interest rates move up, that tends to be a poisonous environment for the commodity complex," said Bart Melek, head of commodity strategy for TD Securities in Toronto.
"The probability of a rate hike has grown, at least according to the Fed Fund futures curve. It's at 60 percent and not that long ago it was around 30."
Read MoreGermany's Degussa: A bull in a bearish gold market
Gold, a non-yielding asset, tends to benefit from a low interest rate environment. It has fallen 5 percent in the six sessions since the Fed's Oct. 28 meeting, when central bankers left the door open to a possible rate hike.
Traders are now awaiting U.S. non-farm payrolls data later this week.
Gold has also come under pressure from outflows from exchange-traded funds (ETFs), with holdings of the largest, New York-listed SPDR Gold Shares, down another three tons on Tuesday.
Interest in gold picked up overnight in some parts of Asia, MKS said in a note on Wednesday.
Silver was down 1.4 percent at $15.05 an ounce, while platinum was down 0.9 percent at $951 an ounce and palladium was down 2.1 percent at $627.75 an ounce.