"I think the stock market's psychological impact — both negative and positive — is almost always underestimated by economists and investors alike," the "Mad Money" host said.
In many declines in the past, it wasn't so much the market event that caused major declines. The hype and fear surrounding the decline prompted volatility in the market.
"I have made no secret of the fact that I think the endless obsession with the Federal Reserve … has caused millions of people to miss this amazing run from the generational bottom seven-and-a-half years ago," Cramer said.
In fact, Cramer refuses to make investments based on what he thinks the Fed will do next. Dissecting every word from Fed officials wastes time for investors trying to find high-quality companies that can dominate a global economy.
So, Cramer warned not to overthink Yellen's speech on Friday. The placid nature of the stock market in the past week could help investors more than they realize, and could resume shortly after Yellen's speech, barring an all-out attack on the market by either major presidential candidate.
"The Fed I've known over the course of my life as an investor is a fickle beast that is often wrong," Cramer said.