U.S. gold settled down $16.40 at $1,329.70 and was last down 1.36 percent at $1,327.80 an ounce, below its 50-day moving average for the first time since June 24. Spot gold was at $1,324.42 an ounce, down 0.97 percent.
Shares of gold mining companies fell 4.5 percent, on pace for a fourth consecutive day of losses for the first time since a 4-day losing streak ending November 6, 2015.
"Gold and silver prices are modestly lower in subdued early U.S. trading Wednesday," Peter Hug, global trading director at Kitco Metals, wrote in a research note. "The outside markets are in a bearish posture for the precious metals on this day, as crude oil prices are lower and the U.S. dollar index is higher."
Previously, the precious metal had been little changed ahead of Yellen's speech, which will be closely watched for further clues on interest rate policy.
Yellen is scheduled to address a meeting of central bankers in Jackson Hole, Wyoming. Recent hawkish comments from policymakers have raised investors' expectations that she might adopt a less cautious tone on rates.