U.S. stock prices held steady, while the dollar rose 0.5 percent against a basket of currencies, pressuring gold, which is priced in the dollar.
"Yellen confirming that the Fed will raise interest rates later this year is nothing new, but nevertheless this helped the dollar to appreciate, which is weighing on the gold price," Commerzbank analyst Daniel Briesemann said.
"We've also had better-than-expected economic data out of the United States, both before and after Yellen's words, so that is also supporting the dollar."
Data released on Wednesday showed U.S. producer prices rose more than expected in June, and that factory activity rebounded in New York state this month.
Read MoreFed's Yellen: Remain on track to raise rates this year if economy evolves as expected
Yellen was scheduled to deliver her second day of testimony on Thursday, this time to the Senate Banking Committee.
European shares steadied at three-week highs, with investors focusing on a crucial vote in Greece over a proposed bailout package to save the country from bankruptcy.
"Greece developments have not had a bullish impact on gold but further glitches in the Greek debt relief process could lend modest support to prices," HSBC said in a note.
The value of New York's SPDR Gold Trust, the largest gold-backed exchange-traded fund, fell as low as $109.58 per share on Wednesday, the lowest since March 2010.
Among other precious metals, silver was down 1.5 percent at $15.12 an ounce, platinum was down 0.5 percent at $1,018.49 an ounce and palladium was up 1.4 percent at $640.75 an ounce.