The Alibaba stake, which Yahoo bought for $1 billion in 2005, could bring in $8.3 billion to $9.5 billion from the IPO. Yahoo would still own a roughly 16 percent stake of Alibaba worth another $26 billion to $27 billion.
Brown cited billionaire activist investor Dan Loeb of Third Point, who made roughly a $1 billion profit after helping push Yahoo's stock price higher beginning in September 2011.
Activists, Brown said, would seize the opportunity.
"There is absolutely no way, in my opinion, they're going to allow this management team, this board of directors to take $6 billion, do a buyback and then have another free $6 billion in cash to experiment," he said. "It's not going to happen."
Brown said that he thought someone would become involved in Yahoo once again by the end of the year.
Read More Yahoo rakes in another jackpot from Alibaba's IPO
Yahoo shares were lower after the start of trading in Alibaba shares.
"When you see weakness on a day like today, I wouldn't be shocked if they're starting a new position or accumulating," Brown said. "And keep in mind, this stock has been an activist target before, to great effect. Dan Loeb made a fortune with it at much lower levels. I really wouldn't be surprised if we saw that by the end of the year. I'm not betting that way with my own money just yet, but I might."
Read MoreDon't buy Alibaba stock: 'Dean of Valuation'
If Loeb doesn't jump on Yahoo, Brown suggested, another corporate raider might be waiting in the wings.
"Do we think Carl Icahn is on vacation right now, or do we think he's watching?"
Disclosure: CNBC has a content-sharing partnership with Yahoo's finance site.