Better than expected jobs and housing data reinforced the view that the U.S. economy is sufficiently robust to warrant rate rises, turning back recent falls for the dollar and pushing 10-year bond yields to their highest since Jan. 3.
A strong dollar makes gold more expensive for holders of other currencies.
Higher interest rates also mean that bond prices fall and yields rise, making such investments more attractive to those looking for safe-haven assets. Though gold is such an asset, it does not offer a yield to investors and costs money to store and insure.
Spot gold fell 0.2 percent to $2,201.76 an ounce, having dropped by 1.1 percent in the previous session, its biggest fall since Dec. 15. It was last up 0.02 percent.
"It looks like the dollar's rise is back on," said Georgette Boele at ABN AMRO. "If you have a higher dollar combined with higher U.S. treasury yields, that's not a good combination for gold prices."
U.S. gold futures were down 0.73 percent at $1,203.1.
Data released before Yellen's speech showed that U.S. consumer prices increased in December at their fastest pace in 2-1/2 years, indicating that inflation pressures could be building.
Yellen will speak again on Thursday, after European markets close, about the economic outlook and monetary policy.
Investors were also looking ahead to the inauguration on
Friday of President-elect Donald Trump, who has mixed promises of tax cuts and infrastructure spending with protectionist statements that have increased demand for gold as a safe haven.
Analysts said that gold's wobble could signal the end of a rally that had lifted the metal by around 8 percent from a mid-December low to an eight-week high of $1,218.64 on Tuesday.
"It's kind of running out of steam unless there's news coming that will bring more money back in," said Tom Kendall at ICBC Standard Bank. "If you see the market close beneath yesterday's low, then you've got to look for a return back to the mid-1,100s at least."
Wednesday's low was $1,201.81.
"We are likely to test below ($1,200) once again and target support at $1,190 and $1,180," analysts at MKS said in a note.