Those comments, and a view that the economy was strengthening, helped send interest rates higher, from a low yield on the 10-year of around 1.6 percent just before the release of the April jobs report in early May, to about 2.2 percent. The 10-year Wednesday was yielding 2.09 percent.
The market has gyrated as traders debate the Fed's next steps.
"If things are too strong they're worried about tapering, and if they're too weak, they're worried about fundamentals. We're caught in a miserable condundrum. We know the punch bowl is going to be taken away or it's going to dry out," Ablin said.
A snap survey by CNBC Wednesday of about two dozen economists, money managers and strategists showed that about 35 percent of them expect the Fed to start tapering its bond purchases in September. The next largest group, 23 percent, expect the Fed to taper in January. Just a fraction, 3.8 percent, see tapering in July, while the same amount expects tapering in 2015 or later.
(Read More: Traders Confused: Is Jobs Market Improving or Not?)
"The market decline has been coincident with a decrease in tapering expectations. The current landscape is obviously incredibly confusing. Japan is a factor," said BTIG chief global strategist Daniel Greenhaus.
Greenhaus said he had been concerned about the lack of commitment among Japanese officials to the Bank of Japan easing programs. "It was a positive to have the world's third largest bank easing incredibly. Late last month they began to waiver, and that was when markets peaked."
Greenhaus said the jobs number may not make that much of a difference to the market and stocks could remain volatile. "I don't think it's going to be enough to break you one way or the other. I don't see a 200,000 plus number," he said. Greenhaus also points out that the ADP number has been lower than the actual nonfarm payrolls number for eight of the last 10 months.
But still, investors are nervous. They pulled $1 billion out of stock mutual funds in the week ending May 29, the second full week this year when the funds saw outflows, according to Investment Company Institute data.
"I think you had a terrific run here on the back of - among other things – certainty, and what's happened in the last few weeks is the level of certainty has given way to uncertainty," said Greenhaus, noting stocks had risen 23 percent since November, before the pullback in the last several days.
"The S&P is a little bit above the trend channel. A move lower of five to seven percent is not out of the question," he said. The last significant pullback for the S&P was in the fall of 2012, when it lost 7.7 percent, but it had also declined nearly 10 percent in the spring of 2012.
S&P/Capital IQ analysts Wednesday raised their 12-month target on the S&P to 1780, from 1670. They said investors should use price declines to add to holdings and cut cash allocations to 10 percent from 15 percent.
The S&P fell to 1608 Wednesday, a 22 point decline and just above its 50-day moving average of 1604. As of Wednesday, it is down 3.7 percent from the all-time high reached on May 21. The Dow lost 216 points to 14,960. The Dow Transports was at 6137, below its 50-day moving average of 6221, and the Dow Utilities fell to 477, below its 50-day level of 513 and 200-day level of 480.
(Read more: Cramer's Lament: 'Alas, the Selling Makes Sense')
"What's scaring a lot of people off is two-fold. One is you've broken a variety of theoretical supports, moving averages in the utilities and all the way through. You broke a pattern in the S&P that takes you down to 1598 to 1600, and the dip buyers are not stepping in," said Art Cashin, director of floor operations at UBS. "Number two is Friday is turning into this (jobs number) is either Armageddon or heaven routine. People are a little wary of doing anything in front of that number."
Another important number for markets is Thursday's jobless claims report, though it will not be a factor in the May employment report. Besides claims at 8:30 a.m. ET, the market is also watching chain store sales.
Wells Capital Management chief investment strategist James Paulsen said a lot of what is going on with the market is an adjustment to higher rates, and he expects that to continue. "I think we could have a five to 10 percent correction from the high levels. I think it's more likely it will be sideways," he said.
"The bigger story isn't the market, it's what's going on in the sectors. You had cyclicals lead off the lows in October to February. Then you had huge underperformance of cyclicals from February to the end of April. Then they kind of surged again during May, and the flip side is you had defensive stocks underperforming, then really outperforming and now they've just got hammered again the last 30 to 40 days, particularly utilities, staples and dividend aristocrats."
Paulsen said the mini corrections across sectors have refreshed them, and has made some sectors, like utilities, attractive. He also likes financial stocks. Financials are one of the best performers of the year, but have taken a new lead position, heading the declines as stocks sold off this week.
"Everybody's paring back on the types of investments that have proved quite rewarding over the last three years. That includes TIPs, munis, a lot of dividend payers, MLPs, REITs, any sector that has benefited from low interest rates is seeing a correction now," said Greenhaus.
HYG, the ishares iBoxx High Yield Bond Fund ETF, and JNK, the SPDR High Yield Corporate ETF are both down about 1.5 percent this week and are negative on the year.
"We need to digest a re-evaluation of equities, and we need to digest a re-evaluation of bonds, and the market needs to digest that ending QE is not that bad, and then (stocks will) resume (higher) next year," Paulsen said. "I think there are opportunities to play. I think the dollar's going to weaken the rest of the year, making offshore more attractive than domestic and making commodities come back in a way that nobody's anticipating."
What else to Watch
The European Central Bank holds a rates meeting Thursday and will hold a press briefing.
Traders are also watching for Japan's auction of $6 billion 30-year bonds, after a weak 10-year auction earlier in the week.