"It was magical, hopefully magical for them, it was magical for all of us to have them there and give them a good chance to have a proper break after eight years of hard work," Branson said.
Upon returning to Washington D.C., Obama said he and his family stayed in the area because his younger daughter Sasha was still in high school.
"We had gotten a house a few blocks from where I used to live," Obama teased, referring to their $8.1 million mansion in Kalorama, located just two miles from the White House.
By the time Obama and his family settled down in their new home, he told Letterman he didn't feel relieved, but rather proud.
"I think there was a sense that I had run the race, I had completed it. I was proud of the work we had done," Obama said. "I was ready for the next stage."
Compared to how quickly life moved for Obama as president, he found that, comparatively, "everything felt like it was moving in slow motion."
He gave the example of his million-dollar Penguin Random House book deal: When Obama's book agent said his publishers were eager to take the next step, Obama was ready to meet the next day. The agent responded by saying, "Oh no, it's going to take two weeks to set it up."
"I had to explain to him where I'm coming from 'right away' means if we don't do something in half an hour somebody dies," Obama said.
Nonetheless, the former president anticipated what lay ahead. "The stereotype of former presidents is you're kind of sitting around your house, waiting for somebody to call and you're kind of lonely and don't know what to do," Obama said. "But the truth is, it felt exciting."
Like this story? Like CNBC Make It on Facebook.