The banks, restaurants and retail stocks should be on fire this time of year. Instead, they are downright toxic, Jim Cramer said.

"They are houses on fire, and there is no letting up in the flames, even as the averages rebounded hard today," the "Mad Money" host said.

As the quarter winds down, Cramer compared each group's progress this year to where investors expected them to go not long ago.

With the expectation that the Federal Reserve would raise interest rates several times in 2016, the banks were supposed to make a fortune. Instead, only one hike came last December and the subsequent rate rise never happened.

Additionally, there have always been a few offsets for banks with weak net interest margins that could augment profits by offering more services to customers and create a steady stream of fees, referred to as cross-selling.

But with Wells Fargo admitting employees opened unauthorized customer accounts for clients to meet sales goals, cross-selling has become a vicious cycle for banks that they must somehow figure out how to stem, Cramer said.

The European banks are also a disaster. London banks finally found their footing, but had core franchises obliterated by the Brexit referendum. Deutsche Bank was in a league of its own, as the Justice Department requested $14 billion for its role in the mortgage crisis, and the bank has only $5 billion to $6 billion in legal reserves.

"Deutsche Bank is in dreamland where it somehow believes things are all hunky-dory, despite relatively low reserves and versus their potentially very expensive legal problem," Cramer said. "Deutsche Bank has historically been totally clueless about the American way of justice when it comes to banks."

Restaurants and retailers should also be doing better, Cramer said. Sonic pre-announced a dreadful number on Tuesday, which came on the heels of a disappointing number from Cracker Barrel. Both companies should have been beneficiaries of lower gasoline prices and higher consumer confidence.

As for retail, Credit Suisse downgraded Macy's to a hold from a buy. Cramer said Macy's has a "miserable trajectory" and represents the rest of retail, with the exception of Wal-Mart and Amazon. Urban Outfitters, L Brands and even Nike are struggling.

"These stocks have become the third rail of investing. In other words, if you can avoid touching them, I recommend it," Cramer said.

Meanwhile technology and semiconductor stocks are one of the only groups that seem to be working right now. Cramer noted that Apple is the star of the group.

So, as the quarter winds down, Cramer found the weakness in banks, retailers and restaurants to be the opposite of what he expected a year ago. This is the reason the market still felt gloomy, even as it looks like many stocks will get out of September unscathed.

Questions for Cramer?
Call Cramer: 1-800-743-CNBC

Want to take a deep dive into Cramer's world? Hit him up!
Mad Money Twitter - Jim Cramer Twitter - Facebook - Instagram - Vine

Questions, comments, suggestions for the "Mad Money" website? madcap@cnbc.com