"We can already feel right now how activity's shutting down for the weekend. It feels like they pulled the plug on the market momentum," said Bank of Tokyo-Mitsubishi chief financial economist Chris Rupkey.
Trading volumes are expected to be sluggish on Friday, and desks should still be relatively lightly staffed, but they will be primed for Yellen's 1 p.m. ET speech on the economy to the Greater Providence Chamber of Commerce. The consumer price index is released at 8:30 a.m. ET, and is expected to show a 0.1 percent gain on the headline, or 0.2 percent, without food or energy.
"Yellen's always important, especially after the headlines in the newspaper, which are powerful, saying that June is off the table," said Rupkey.
If the CPI is as expected, it would be the fourth month in a row of 0.2 percent inflation, excluding food and energy, he said. "I'm pretty bearish on interest rates, and certainly would like to see lift-off early and rates higher. I'm afraid for this number tomorrow. It might be in the dove's camp. It might be 0.1," said Rupkey.
CPI is important since it is expected that inflation will continue to be low, while the Fed has been looking for signs it will start to pick up. "They seem to want to believe the outlook for inflation. They have to be assured the outlook for inflation is such that it will get to 2 percent sometime in the intermediate term. If we get a tenth on the core rate, that's going to let the air out of that balloon," Rupkey said.
After CPI, traders will quickly turn their attention to Yellen to see if she characterizes the economy in any new way, which could serve as a clue as to the Fed's thinking on rate hikes.
Rupkey said Yellen may make a comment on the labor market or the unemployment rate, since Rhode Island has one of the highest unemployment rates in the country.
"I don't think there will be any major fireworks with either one. CPI is obviously going to be on the tame side. It's not going to be a terrible number. It will show a little bit of inflation but nothing to concern the Fed," said Justin Lederer, interest rate strategist at Cantor Fitzgerald. "Yellen, I think she's expressed her opinion. I don't think she really wants to rile the market The June rate hike is pretty much off the table, and I think September is where the target meeting is right now, and we'll continue to watch the data."
Read MoreDow divergence spooks traders
Lederer said the bond market is beginning to calm down after a recent volatile run higher in yield. Disappointing existing home sales data helped send yields lower Thursday with the 10-year note reaching 2.18 percent. "If today means anything, I think the market has found some stability…. It's really been off the heavy corporate issuance. If that calendar slows down, Treasurys may try to find stability. It's easier said than done. A month ago, we were in a 15 basis point range in the 10-year note, and people thought we wouldn't break 2 percent…. I think it's a volatile market and it will continue to be," he said.
A surge in corporate debt issuance helped push May's investment grade total to $136.6 billion, just $13 billion shy of an all-time high monthly total of $149.8 billion, reached in March, according to Reuters.
Stock traders will be also watching Yellen for any further comments on the stock market, after she said she believed it was overvalued.
"What's she going to say? Biotechs are overvalued?" quipped one stock trader.