Levinsohn said that at this "moment of truth," some investors (including Starboard Value's Jeffrey Smith) and board members may begin to evaluate the extent of Yahoo's gains under Mayer's leadership.
"She was given a very long leash to build that company, it's now almost three years. You can look at the performance, it has not gotten to where a lot of people thought it would get to," he said.
Yahoo's stock has risen about 190 percent since Mayer took over in July 2012.
Read MoreYahoo's Mayer saves job with Alibaba spin: Experts
Levinsohn, who is the current executive chairman of Scout, said he views Yahoo's immediate future as "a bit binary."
"If Marissa can get this company growing in an effective way, I think it's a great outcome for the company, she can ride off into the sunset as a hero, and everybody's happy because the stock will likely go up," he said.
But on the other hand, he said, "If the company and the stock continues to struggle in terms of growth...this is a likely target."
If private equity ends up buying the company, Levinsohn said, "you would probably see some dramatic changes," which could include Mayer leaving the firm.
Read More Starboard letter warns against further Yahoo acquisitions
Although he had kind words for Mayer's energy and expertise, Levinsohn admitted that she had a challenging job.
"One person cannot change Yahoo...and the stock growth, based on most reports, is really keyed to Alibaba," he said, adding that "it is a really really tough business to go run and grow."
The company did not immediately return a request for comment.