Drugstore chain Walgreens Boots Alliance, with a market capitalization of $64 billion, compared with General Electric's $110 billion market cap, was not the obvious choice, said Cashin, managing director of UBS Financial Services. But the addition of a tech giant, such as one of the FAANG stocks, would have been a mistake too.
"The problem with replacing GE, if you put one of these mega monsters in, an Amazon or something like that, it would absolutely swallow the whole average, because it is based on price, not on capitalization," said Cashin, who has worked at the New York Stock Exchange for more than 50 years.
In fact, all of the tech giants are significantly higher: Amazon has a market cap of more than $850 billion. Google is valued at approximately $830 billion. Facebook has a market cap of about $587 billion, and Netflix is about $181 billion.
"The primary concern was not to bring in any tech that looked like it might wind up swallowing or distorting the Dow," he said. "If the Dow began to part company with all the rest of the indices, you'd have people questioning its validity."
On Tuesday, GE was removed from the Dow, to be replaced by Walgreens starting June 26.
Still, the move is not a surprise to everyone.
GE has had a bad year: It was the worst-performing stock in the index, and shares have dropped more than 55 percent in 12 months. Worse yet, now that the industrial giant is out of the Dow, some index funds are no longer required to own the stock.
Former GE Power Systems President and CEO Bob Nardelli said GE was "weighting down the Dow."
"It's been rumored for a while," Nardelli said Tuesday on "Fast Money." "And when people use that as a barometer for investment, I'm not surprised that they decided to transition GE out."
General Electric's stock was down about 0.4 percent Wednesday 2 p.m. ET on the New York Stock Exchange.
The Dow consists of 30 of the largest publicly traded companies in the U.S. GE was one of the original components of the blue-chip index when it was created in 1896 and has been listed on the index for more than 100 years.