President Carter, along with Kofi Anan and Archbishop Desmond Tutu are original members of the group, which was founded by Nelson Mandela.
"We just wanted one American as a founding member of The Elders. And we obviously didn't want the Elders to be dominated by any particular country. Nelson Mandela felt very strongly that the one American that epitomized everything that The Elders were to stand for was President Carter," Branson told CNBC's Tania Bryer.
Read MoreCNBC Meets: Sir Richard Branson
Branson said Carter, who turns 90 this month, was incredibly unfairly judged during his time at the White House and said the President struggled because he was "ahead of his time".
"He put solar panels on top of the White House, Reagan immediately took them down when he left the White House. I think if it hadn't been for the Iran hostage situation he would have had a second term and I think most likely would've been as well-respected for his time in office as for his time outside the office," he added.
CNBC Meets: President Jimmy Carter will air on 1 October at 23:00 CET