Investors were more familiar with Levinsohn and investors will want to see him stay with the company, Gillis said.
"If they can keep Ross in the company, that will be a real coup," Gillis said.
Leadership at Yahoo has been rocky over the last two years, with multiple people taking the chief position at the company only to have limited success in turning the company around.
Former Yahoo CEO Scott Thompson led the company for less than six months when he resigned in May over a controversy regarding his academic credentials.
During Thompson's time as CEO, the company made moves to restructure.
Yahoo announced in April that about 2,000 employees would be let go and a reorganization plan was laid out that was supposed to help the company operate more efficiently. The company said the changes would help curb Yahoo's costs by $375 million annually.
Tim Morse served as an interim CEO before Thompson, following Carol Bartz time as CEO.
Bartz served as CEO for two years (2009-2011) and although she also employed lay-offs and attempted to restructure, she had little success. She was fired in September 2011 for not turning the company around.
Yahoo's stock rose in after-hours trading, topping $16 a share.
Disclosure:
CNBC and Yahoo! have a business alliance to share and co-produce editorial content.