Flannery, 55, began his career at GE Capital in 1987, where he focused on evaluating risk for leveraged buyouts. He has led GE's equity business in Latin America and the overall GE Capital
business for Argentina and Chile.
Flannery also spearheaded the turnaround of GE Healthcare, increasing organic revenue by 5 percent and margins by 100 basis points in 2016, GE said.
"Today's announcement is the greatest honor of my career," Flannery said in a statement. "I am privileged to have spent the last 16 years at the company working for Jeff, one of the greatest business leaders of our time."
Immelt, 61, will remain chairman of the board through his retirement from the company on Dec. 31.
Additionally, Chief Financial Officer Jeff Bornstein will become vice chairman of the company, and Kieran Murphy has been appointed CEO of GE Healthcare, succeeding Flannery. Murphy is currently president of GE Healthcare's life sciences division.
GE said these moves are part of its succession plan, which has been in place by the company's board since 2011.
CEO Immelt led GE through a portfolio transformation, the 9/11 tragedy, the financial crisis and oil price dynamics, among other things, the company said. "Immelt's vision has positioned GE for the future and greatly strengthened the GE culture."
Immelt, who succeeded Jack Welch in 2001, also aided in restructuring GE Capital, shifting the company's focus from finance to more manufacturing.
"Jeff brought his best every day for 16 years," Welch said in a statement Monday. "I wish him the very best in the many good years he has ahead."
Shareholders, though, have not been happy with the company's financial performance over the past decade under Immelt's leadership, Barclays' Davis told CNBC. The CEO was struggling to connect with investors, and now many are expecting "fairly dramatic changes" under Flannery, he said.
Several big names on Wall Street weighed in Monday on Immelt's departure.
"Jeff has dramatically transformed General Electric over the past decade — adapting during turbulent times and positioning the company extremely well for many years to come," JPMorgan's Jamie Dimon said in a statement. "And he has done so with unflinching integrity and a commitment to executing on a bold vision for future success."
In his fifth tweet, Goldman Sachs CEO Lloyd Blankfein shared congratulations.
— Reuters contributed to this report.
Read more: Jeff Immelt's tenure at GE was an 'unmitigated disaster for shareholders,' analyst says