In a move announced on Thursday, he's going from his former position at the Federal Reserve to Wall Street as a senior adviser at Citadel. The latter is what has "Fast Money" trader Guy Adami—and a number of other Street watchers—outraged.
The $25 billion hedge fund, Citadel, in a statement said, "Dr. Bernanke will consult with Citadel teams on developments in monetary policy, financial markets and the global economy." Adding a note from its founder and CEO Ken Griffin, "He has extraordinary knowledge of the global economy and his insights on monetary policy and the capital markets will be extremely valuable to our team and to our investors."
Read MoreBen Bernanke memoir will be titled: 'The Courage to Act'
Bernanke "was a hero for a month, [and now] he's going to go down as one of the most vilified people of the 21st century. Mark my words," the trader added.
In an interview with Andrew Ross Sorkin, co-anchor of CNBC's "Squawk Box" and a columnist for the New York Times, Bernanke said he understood the concerns about going from Washington to Wall Street. He said he decided in Citadel because the hedge fund "is not regulated by the Federal Reserve and I won't be doing lobbying of any sort."
He also said banks had approached him about jobs but he declined because "wanted to avoid the appearance of a conflict of interest" by working for an institution the Fed does regulate.
Bernanke is not the first and likely won't be the last federal worker to jump to Wall Street.
In 2008 after handing over the reins to Ben Bernanke, Alan Greenspan joined hedge fund Paulson & Co. as an adviser. And just last month, Ex-Fed Governor Jeremy Stein joined $20 billion hedge fund Blue Mountain Capital Management as a consultant.
"He shouldn't have been allowed to leave the Fed, number one," Adami stated. "He should have saw [quantitative easing] through, in my opinion, and for him to go to a place that can take advantage of the information that he has privy to, it's just wrong."