"If any new taxes are levied on me, or my company, as our current president plans," Siegel wrote in his memo, "I will have no choice but to reduce the size of this company. Rather than grow this company I will be forced to cut back. This means fewer jobs, less benefits and certainly less opportunity for everyone."
But today on CNBC's Closing Bell, Siegel said that he's keeping his employees, and giving them raises to try to double sales this year.
"It was not a political ploy," he told CNBC. "I never said I was going to fire anyone if President Obama was re-elected. What I said was that iF his policies caused the company to cease making money, to cease expanding, I'll most likely have to lay off people down the road. Time will tell." (Read more: CEO to Workers: You'll Likely Be Fired If Obama Wins)
Siegel's company, Westgate Resorts, is the largest time-share company in the world. While Siegel says Westgate is more profitable than ever, it's sales are down by half and its workforce down by 7,000 people since the start of the financial crisis.
Siegel and his wife, Jackie, have re-started construction on Versailles, their house near Orlando, Florida that's slated to be among the largest in the country, at more than 90,000 square feet. The story of the Siegels and their house was first told in The High-Beta Rich.