"You've got to separate a service you love from an investable company," Brown, who has 1.1 million Twitter followers, said on CNBC's "Fast Money Halftime Report."
Brown said he took a small profit Tuesday when he completely exited his position. He said he has bought shares at various price points, some when the stock traded in the $20s and up in the $40s around its IPO.
"I have been very wrong on Twitter," said Brown, who said he had planned to gradually sell out of Twitter in 2020. He runs Ritholtz Wealth Management, a New York City-based investment advisory firm. He also writes "The Reformed Broker" blog.
But Brown said Dorsey's decision to move to Africa for a few months next year, plus Dorsey's decision this fall to ban all political ads rather than fact-check them on the social-media platform, accelerated his decision to sell.
Brown called Dorsey, who also is CEO of Square, a "half CEO," and said "I really don't understand what's going on with the governance of this company."
The decision to ban political advertising heading into a presidential election cycle — in contrast to Facebook's decision to not fact-check ads — is in direct opposition to Twitter's best characteristic, Brown argued.
"This is like the only reason to own the stock. This is what Twitter does best: It influences politics," he added. "You might not like the direction in which it does so, but it's undeniable how influential the platform is."
President Donald Trump has 68.1 million Twitter followers and often uses the platform as a bully pulpit.
The stock was relatively steady, trading just below $32 intraday Tuesday.
Twitter went public on Nov. 7, 2013, and ended its first trading day at $44.94 — up nearly 73% from its $26 offering price.
The stock shot up in the $70s per share shortly after its IPO. But it traded down in the teens in 2016 for a while. It got back in the $40s last year and then bounced around to lower levels.
In a year when the S&P 500 was up 30% and rival Facebook was up 56%, Twitter shares were just 11.7% higher in 2019.
Twitter, from a business standpoint, is "not in trouble," Brown said.
"They're OK. They're profitable," he said. "But they're not serious about getting to the level that Facebook has gotten to and that LinkedIn, under Microsoft, has gotten to."
Brown did not rule out investing in the company again, potentially if an activist investor gets involved.
"I still think the company is going to do well. I just don't think it's a great stock to be in right now," Brown said. "There are way too many opportunities."
Brown did not mention any opportunities by name — though he did say there are no attractive social media investments outside of Facebook.
He said going from investing in Twitter to Snapchat-parent Snap would be akin to going from the "frying pan to the fire."
Despite an initial pop around its March 2017 IPO, Snap stock has been dog for early investors, trading in the single-digits last year and still below its $17 offering price.
It is worth noting that Snap shares were up nearly 200% in 2019 from those very low levels. The stock was trading just above $16 on Tuesday.