European Central Bank President Mario Draghi is also expected to speak at Jackson Hole, at 2:30 p.m. EST. "Draghi is going to be a wild card. What's he going to do? Europe is weakening," Cashin said.
The market is looking for Draghi to explain whether Europe will carry out its own version of quantitative easing, or asset purchases.
"He's going to have to explain it. A lot of people think he's handcuffed. Whatever he says will be a surprise. Either he admits he's handcuffed, or he explains it."
Read MoreFed's Williams: Summer rate hike 'reasonable'
Scott Redler, partner at T3Live.com, said the market was also boosted by the bank stocks. The S&P financial sector was the best performing major sector, up 1.2 percent Thursday, in its best day since May 7.
"You had technology acting a little weird. High beta names had no power. It was the banks that carried the baton today," said Redler, who follows short term techincals.
"Most thought it could happen going into Jackson Hole, and the Fed minutes but we weren't prepared for it. When Bank of America opened up on the settlement talks and squeezed every short, people started putting money into the banks. If the banks follow through (Friday) and don't give much back, the banks would be the group that takes the S&P to 2000."
Redler said if Yellen's comments push the S&P above 2,000, it may not go much further without taking a breather.
Read MorePlosser: Risk rates may need to rise 'very quickly'
"At this point, it feels like you're chasing the market. The bulls are kind of scared to buy and shorts aren't really working. Plus everyone's on vacation–and it still goes higher. It's a weird dynamic. A better dynamic might be to be on a beach," he said.