"Unfortunately they seem in muddled message mode right now," said McCarthy. Markets will also digest the comments of more hawkish Kansas City Fed President Esther George, who was to speak Thursday evening. George is the one member who dissented at the last central bank meeting, when it voted to keep rates unchanged.
There will also be comments to consider from Yellen herself, plus her last three predecessors, at an unprecedented panel discussion Thursday evening.
Yellen said Thursday that the U.S. economy is strong and said it is not a bubble economy. She reiterated that the Fed is taking a cautious approach on raising rates and said the Fed's rate hike in December was not a mistake.
Her comments Thursday evening were viewed as more optimistic than in her appearance last week, and they helped support risk markets, including oil. Now the question is which way will the next speakers lean.
Read MoreYellen: U.S. is not a 'bubble economy'
Some of the more hawkish Fed presidents have been calling for rate hikes, but the markets snapped to attention when Yellen delivered a very dovish speech last week, detailing risks to the economic forecast and pointing to a slow and cautious path for the central bank. Those comments reinforced the dovish message from the Fed's March meeting, and markets have adjusted.
Read MoreThe big S&P prediction that 'no one believes'
The currency market in particular has responded to the Fed, with the dollar losing ground. The greenback has declined more than 5 percent against the yen since the March 16 FOMC meeting.
Dollar/yen kept gathering steam Thursday, with the yen at 108 against the dollar after breaking 110 a day earlier. The move has been perplexing since Japan has gone to negative yields, and its economy is weak.
The S&P 500 fell 1.2 percent to 2,041, just under a significant support level, while Treasury yields moved lower. The 10-year yield edged down and was at 1.68 percent in late trading. West Texas Intermediate crude futures fell 1.3 percent to $37.26.
"I would say the spike in the yen put a dagger in the heart of the financial community. They're afraid that (Japanese Prime Minister Shinzo) Abe and (Bank of Japan Governor Haruhiko) Kuroda are going to do something drastic, like taking rates much more negative. They don't think they're going to be able to do that for long," said Art Cashin, UBS director of floor operations at the NYSE. "It will put more pressure on the banking system if either Japan or Europe gets much more negative in their rates."
Read MoreBillions of dollars bail on stocks and buy bonds
The move in the yen has been spooking some in the markets, who see it as a currency that has usually benefited during a flight to safety.
"I think investors are sort of trading on a slower growth tilt," said Jack Ablin, CIO of BMO Private Bank. "When you add up the lower stocks, lower interest rates, yen higher, gold higher, oil lower, it's a complete setup."
Read MoreThe end of short covering and 'Fed put' spells market trouble
Ablin said the stock market is seeing more stretched values after the recent runup, and it faces a tough earnings season, beginning next week.
Besides Fed speakers Friday, there is wholesale trade data at 10 a.m. EDT.
Read More How to invest in Jamie Dimon's market forecast