The Dow dropped 40 to 17,678, erasing a triple-digit loss, and the S&P 500 was off nearly 5 at 2,056. "I think it's kind of normal profit-taking. I think eventually the economic data is going to rebound and the corporate profits will come in OK, and that will take the market higher," said Andrew Burkly, head of institutional portfolio strategy at Oppenheimer Asset Management.
The market has been worried in the past several sessions about forecasts for a decline in S&P 500 earnings this quarter—the first in six years. The strong dollar and falling oil prices are blamed for the dip in corporate profits.
Burkly said there is a lack of catalysts for the market to consider in the next week, and that could create a vacuum for more selling. "The market's felt a little heavy," he said.
Treasury yields, however, were higher as bonds also sold off Thursday, and the 10-year crossed above the key 2 percent level for the first time since March 18, the day of the Fed's meeting.
"We're talking Fed, but I don't think we're trading Fed," said David Ader, chief Treasury strategist at CRT Capital. "I think we're dealing with a positioning story and repatriation story. ... We've got the Fed priced in and we're going to see volatility around month-end, quarter-end."
Read MoreReversal today? If so, it could be temporary
Ader said the market also was responding somewhat to hedging around the $5 billion offering from German development bank KfW. He also said technical factors were behind the move up in yields, and a lack of liquidity in the bond market exaggerated moves.
"Let's get use to this. The dealer community is not about to expand. You're not getting more balance sheet traders. As we get to these moments, whether it's from quarter-end or positioning extremes, you're going to find, like during the taper tantrum, a lot more of these moves that defy fundamental logic and then have a life of their own," he said.
But traders will watch Yellen regardless, as the market struggles to figure out the timing of that first rate hike. "She's not going to tell us more than we got last week. She got her opportunity and not a lot has changed, unless she tells me it's June, July or September," said Ader.
The Fed did spur a weeklong selloff in the dollar that abated somewhat Thursday, as the dollar index turned higher.
Read MoreFed policies have cost savers $470 billion: Report
"Markets have still been digesting last week's Fed announcement and realize they're not in a rush to hike and if they're going to do it, they're going to be gradual about it because they don't want to risk the recovery with premature hiking," said George Goncalves, head of rate strategy at Nomura.
Another Fed luminary, Vice Chairman Stanley Fischer, speaks on regulatory issues in Frankfurt at 6:30 a.m. Friday. Earlier this week, Fischer spoke before economists in New York and said he expected the U.S. central bank to raise rates before the end of the year. He also emphasized that the path of future rate rises will not be smooth but offered no new insights.
Yellen definitely gets top billing Friday. "The conference is on the new normal, so it's a relevant topic. I suspect (Yellen's) discussion will be broader and a bit more academic than what the market wants to hear," said Amherst Pierpont Securities' chief economist, Stephen Stanley. "I don't know that she has anything new to add after a week."
"The Fed says they're data dependent. Now the market has to pay attention to the data," Stanley said. "Starting in June, anything's on the table and it depends on how the data plays out."
Friday's data include a final reading of fourth-quarter GDP, expected to rise to 2.4 percent from 2.2 percent. The 8:30 a.m. ET number will be followed by consumer sentiment at 10 a.m.
Read More'Three black crows' could spell doom for the market
"It might help to focus people on the first-quarter numbers and the first-quarter numbers are starting to feel like a replay of last year," said Stanley. "We're going to see a very slow first quarter, and I don't think that's been internalized. The median forecast for Q1 is still around 2 percent."
Data this week included a disappointing durable goods report which resulted in economists slicing forecasts for the first quarter. According to a CNBC/Moody's Analytics survey, economists now expect growth of 1.8 percent.
"I'm down at a half percent," Stanley said. "Based on what we know today, I wouldn't rule that out."
While markets struggle with concerns about the weak economy, the debate on Wall Street has been when and how much will growth will pick up. That debate goes directly to the heart of the discussion on when the Fed will raise interest rates.
Read MoreWar in Yemen: What it means—and doesn't mean
"I think we'll get a pretty big snapback," Stanley said of the second quarter.
The market is already focusing on next week's jobs report, expected to be released on the Good Friday holiday when the stock market is closed.
The jobs number should be fairly strong, and it is the one data series the Fed relies on that has shown consistent strength. "I'm at 275,000. I think we've turned the corner. I think the underlying trend is very strong," Stanley said.