Sessions' chief of staff, Matthew Whitaker, will serve as acting attorney general, Trump announced.
Whitaker also will assume oversight of the ongoing investigation by special counsel Robert Mueller into Russian interference in the 2016 presidential election, and possible collusion by Trump's campaign in that meddling, according to the Justice Department.
Whitaker, who has publicly criticized the Mueller investigation, by law can serve as acting AG for a maximum of 210 days.
He will have the power to fire Mueller "for cause" as outlined under rules governing the special counsel's office, if such cause is found.
Deputy Attorney General Rod Rosenstein up till now has had oversight over the Mueller probe, as a result of Sessions' move to recuse himself from the investigation in 2017.
Sessions, 71, had been repeatedly blasted by Trump for his recusal.
Trump has said that moment opened the door to special counsel Mueller's probe, which the president has repeatedly called a "witch hunt."
Sessions' resignation was expected to happen sometime after Tuesday's midterm elections, particularly given the drumbeat of the president's repeated criticism of the attorney general.
Bob Woodward's recently published book about the Trump administration, "Fear," says that the president had called Sessions "mentally retarded" and a "dumb southerner." Trump has publicly claimed, "I said neither" about Sessions.
"I don't have an attorney general," Trump told The Hill in an interview with that news site in September.
But the abruptness of the move, less than 24 hours after the close of the polls Tuesday, stunned Trump's closest allies both inside and outside of the White House.
"I didn't know this was coming, especially so soon after the midterms," one source said on the condition of anonymity.
Hours before the resignation was announced, Trump was asked about Sessions' future in the administration.
"I'd rather answer that at a little bit different time," the president responded.
Trump's press secretary, Sarah Huckabee Sanders, said the White House received a resignation letter from Sessions earlier Wednesday and Trump accepted it.
Sessions' letter to Trump, which is not dated, begins by saying, "At your request, I am submitting my resignation."
A spokesman for Mueller's office declined to comment when contacted by CNBC about the resignation.
But Rep. Nancy Pelosi, D-Calif., who is the odds on favorite to become the next speaker of the House of Representatives, called Sessions' "firing" a "blatant attempt" by Trump to undermine Mueller.
Sen. Chuck Schumer, a New York Democrat and Senate minority leader, said, "Protecting Mueller and his investigation is paramount."
"It would create a constitutional crisis if this was a prelude to ending or greatly limiting the Mueller investigation and I hope President Trump and those he listens to will refrain from that," Schumer said.
Rep. Jerry Nadler, D-N.Y., the incoming chairman of the House Judiciary Committee, in a tweet wrote: "Americans must have answers immediately behind" Trump removing Sessions from the Justice Department.
"Why is the President making this change and who has authority over Special Counsel Mueller's investigation? We will be holding people accountable," Nadler wrote.
— Additional reporting by Kevin Breuninger and Brian Schwartz of CNBC.
Read Jeff Sessions' resignation letter here.