It seems Zimmer wanted to remain influential in the company's path forward, and letting go of a company he founded certainly couldn't be easy after 40 years.
Zimmer once sold raincoats out of the trunk of his car, and is now the face of a brand that has 40 percent market share in U.S. tuxedo rentals and around 28 percent market share in men's tailored clothing, according to industry estimates.
Although the founder's influence has been scaled back recently, some think it would take a drastic move, at a significant time like a shareholder meeting, for the board to finally cut ties.
It couldn't have been easy, the board isn't exactly 100 percent independent. Zimmer built his board over time, to a group composed—at least until recently—of his allies.
One board member, best-selling author and physician Deepak Chopra, issued a statement expressing his support for CEO Doug Ewert and the company's management team. "As a member of the board of the Men's Wearhouse, I am very excited about the future of the company," he said.
Ending ties with Zimmer won't be very easy. He is the company's largest individual shareholder with a 3.5 percent stake.
And since his likeness is so connected to the company, Men's Wearhouse has to pay for its use even in the event of a termination. According to SEC filings, Men's Wearhouse is required to pay Zimmer or his estate $250,000 per year for four years for the continued license. Once those four years are up, it will have the option to continue to license and subsequent payments for its use.
However, Jaffe said that the company has been evaluating the effectiveness of Zimmer in his brand messaging, concerned it may not be resonating with the ever-important millennial consumer.
Jaffe is one of six analysts who cover Men's Wearhouse, and he has reiterated his "buy" rating on the stock. Men's Warehouse shares did open lower on the news Wednesday, but the stock eventually retraced some of its early losses.
In a press release, Men's Wearhouse said, "the board expects to discuss with Mr. Zimmer the extent, if any, and terms of his ongoing relationship with the company."
Ouch.
—By CNBC's Courtney Reagan. Follow her on Twitter @CourtReagan.