The S&P 500 closed down 5 points Wednesday at 2,170. That decline wiped out its gains for the month and the S&P 500 ended down 0.1 percent, after being higher for a good part of August. August was the first losing month for the S&P 500 since February. The S&P energy sector was down 1.4 percent Wednesday, but financials were up very slightly at 0.1 percent.
"Oil is dragging the market down, and I think the fear is building that if the employment number is good, the Fed will pull the trigger," said Steve Massocca, managing director with Wedbush Securities.
Oil fell Wednesday after the government reported large surprise builds of U.S. crude inventories and distillate stockpiles. Oil supply grew by 2.3 million barrels last week. The government also said that 19.5 percent of crude equivalent production has been shut in the Gulf of Mexico due to tropical storms, though that has not been a factor in oil prices.
The dollar has also been a weight on oil, though the dollar index turned lower Wednesday. OPEC meets with other producers at the end of September, and key members like Saudi Arabia have said there could be discussions about how to stabilize prices.
"I think the idea they will have a production freeze agreement is dwindling," said Gene McGillian of Tradition Energy.
But it is the Fed hanging over all markets, including energy, and oil could take another leg down if it looks like the Fed will hike rates.
Mark Zandi, chief economist at Moody's Analytics, said he expects 150,000 jobs for August, in part due to a seasonal drag that has affected August employment reports. The consensus forecast is for 180,000 jobs, down from July's 255,000.
"Subtracting from these technical issues, I think job growth is somewhere between 175,000 and 200,000," he said. ADP payroll data showed 177,000 jobs for August.
Zandi said a number in line with the Wall Street consensus would probably not prompt the Fed to raise rates in September. December is widely viewed as the most likely time.
"I think they can wait. I think they're very cautious. This FOMC is particularly cautious, and I don't think they'll move. I think they could have a window where markets are very calm, volatility is low. They have an opportunity. They should, but I just sense they are very cautious and will wait," said Zandi. However, if the report comes in sharply higher than expected, as many traders are discussing, then the Fed could be pushed.
"I think you need something like 200,000 and a 4.8 percent unemployment rate. I think if you got those numbers, that could be enough to convince enough people to move," he said.
September kicks off with a big batch of data Thursday. Initial claims and productivity and costs are released at 8:30 a.m. EDT Thursday. Manufacturing PMI is at 9:45 a.m. EDT and ISM manufacturing is at 10 a.m., as is construction spending. Automakers release vehicle sales for August on Thursday morning.
Stocks are heading into the seasonally tough month of September. According to Stock Trader's Almanac, September is the worst performing month of the year for major stock indexes. Since 1950, the S&P 500 has seen an average September decline of a half percent, though it has done better in presidential election years, the almanac said.
Analytics firm Kensho looked at more recent time frames for August. Since 2010, in the post financial crisis years, the S&P 500 has been up half the time for an average 0.3 percent gain in September, but over the last 20 years, the S&P fell 0.5 percent on average in September but was higher 55 percent of the time.
As for performance in September and beyond, the period from Aug. 31 to Dec. 31 has been mostly positive for stocks on average going back to 1996. According to Kensho, the S&P 500, Dow and Nasdaq are up 80 percent of the time. The S&P 500 averages a return of 4.3 percent during that time. Energy has been the worst performing sector in that time frame but is up 2.4 percent.