"This was not an easy decision for me to make, and in truth it will never feel like the right time to leave a company like Sainsbury's," King said in a statement.
"I am confident that under Mike's leadership the business will go from strength to strength."
During King's 10-year tenure at Sainsbury's, turnover increased by 67 percent and operating profit rose by almost 60 percent.
Coupe, 53, will become head of a grocer that is battling with Asda (owned by Wal-Mart) to become the second-biggest food retailer in Britain after Tesco.
In the 12 weeks to January, 6, Sainbury's had 17.1 percent of supermarket share in the U.K., according to Kantar Worldpanel research firm, behind Asda's 17.5 percent and Tesco's 30.4 percent.
The group's Chairman David Tyler said King had "reshaped" the company during his decade at the helm. He added that he was "delighted" to appoint Coupe his successor.
"No one knows Sainsbury's – or the industry – better than Mike," Tyler said in a statement. "He has worked hand-in-hand with Justin over the past decade and has a proven track record of success making him the natural choice to take the company forward."
In a statement, Sainsbury's said King was eligible for payment of an annual bonus and deferred share award for 2013/14, but had waived a cash severance payment, worth up to £1.7 million ($2.82 million). Coupe will receive an annual salary of £900,000.
Phil Dorrell, director of retail consultants Retail Remedy, said that Coupe should aim to ride out the momentum provided by King.
"If it ain't broke, don't fix it - and right now Sainsbury's is by no means broke. For Mike Coupe, in the near-term at least, consistency is key," he said.
"The markets will have their eye on Sainsbury's in the second half of the year and be closely monitoring the performance of Coupe."