Fmr. Yahoo CEO: Marissa Mayer may have to go

Yahoo's moment of truth
Levinsohn: Incredible runway for Yahoo
Yahoo should focus on improving operations: Starboard

Yahoo CEO Marissa Mayer may have to leave the tech giant soon, according to a former head of the company who left after losing out to Mayer for the job three years ago.

Former Yahoo interim CEO Ross Levinsohn suggested that the future of the firm will either see it being sold to another entity, or it will mean Mayer is forced to leave her position—or maybe both.

"I think it's completely in her hands," he said of Mayer's future as Yahoo's CEO, in a Wednesday interview with CNBC's "Fast Money: Halftime Report." It all depends, he said, on how the company's stock fares after it spins off its remaining stake in Chinese e-commerce giant Alibaba.

Ross Levinsohn
Patrick T. Fallon | Bloomberg | Getty Images

Levinsohn said that at this "moment of truth," some investors (including Starboard Value's Jeffrey Smith) and board members may begin to evaluate the extent of Yahoo's gains under Mayer's leadership.

"She was given a very long leash to build that company, it's now almost three years. You can look at the performance, it has not gotten to where a lot of people thought it would get to," he said.

Yahoo's stock has risen about 190 percent since Mayer took over in July 2012.

Read MoreYahoo's Mayer saves job with Alibaba spin: Experts

Levinsohn, who is the current executive chairman of Scout, said he views Yahoo's immediate future as "a bit binary."

"If Marissa can get this company growing in an effective way, I think it's a great outcome for the company, she can ride off into the sunset as a hero, and everybody's happy because the stock will likely go up," he said.

But on the other hand, he said, "If the company and the stock continues to struggle in terms of growth...this is a likely target."

If private equity ends up buying the company, Levinsohn said, "you would probably see some dramatic changes," which could include Mayer leaving the firm.

Read More Starboard letter warns against further Yahoo acquisitions

Although he had kind words for Mayer's energy and expertise, Levinsohn admitted that she had a challenging job.

"One person cannot change Yahoo...and the stock growth, based on most reports, is really keyed to Alibaba," he said, adding that "it is a really really tough business to go run and grow."

The company did not immediately return a request for comment.