She also noted that the Fed was watching the issue closely.
"I would highlight that equity market valuations at this point generally are quite high," Yellen said, according to Reuters. "There are potential dangers there."
Stocks, which had turned lower after a strong open, fell further following Yellen's comments.
"She's really raising those questions that valuations are pretty full. I agree with that," said David O'Malley, CEO of Penn Mutual Asset Management.
Yellen also made note of the risks to open-ended mutual funds, Reuters reported, particularly dangers to liquidity if redemptions rose.
"It's more likely that Janet Yellen's comments spilled over into the downturn. Yellen's comments are in line with things she said in the past on overstretched valuations," said Ben Garber, capital markets economist at Moody's Analytics.
Earlier Wednesday, Yellen said the Fed and other banking regulators have made significant progress in correcting flaws in the financial system that triggered the worst banking crisis in seven decades.
Banking regulators are remaining "watchful" for any areas where further reforms may be needed, she said in remarks at a financial conference.
Yellen cited the need to address the problem of "too big to fail"—the perception among investors that some institutions are so large that the government will step in and save them if they get into trouble.
She said the Fed and other regulators are taking steps to make sure that the collapse of even very large banking institutions can be handled in ways that don't jeopardize the stability of the entire system.
Lagarde told the group that a recent IMF report found that risks to financial stability around the globe are rising with increasing risks at non-bank financial institutions and in emerging market countries.
"We need to build a financial system that is both more ethical and oriented more to the needs of the real economy—a financial system that serves society and not the other way around," Lagarde said.
Yellen said a well-functioning financial sector promotes job creation, innovation and economic growth but that problems arise when the incentives become distorted, prompting bank executives to pursue risky strategies to increase profits.
"Unfortunately, in the years preceding the financial crisis, all too many firms took on risks they could neither measure nor manage," she said.
"The result was the most severe financial crisis and economic downturn since the Great Depression," the Fed chief said, noting that 9 million American lost their jobs and roughly twice that many lost their homes.
—CNBC.com and wires contributed to this report.