The dispute between Twitter and the U.S. intelligence industry that broke into public view this week actually began as long ago as last fall, sources familiar with the matter tell CNBC.
The Wall Street Journal reported this week that Twitter has cut off U.S. intelligence agencies from a service that sifts through the entire output of Twitter's social media postings. That comes as Silicon Valley and the U.S. government have been engaged in a heated dispute over the degree to which American tech companies should cooperate with U.S. law enforcement and intelligence agencies, such as the FBI and CIA.
It has not been clear exactly which entity in the vast U.S. intelligence apparatus was involved in the dispute with Twitter, but sources tell CNBC that it was a division of the CIA known as Open Source Enterprise. According to the CIA's website, that unit is a part of the CIA's directorate of digital innovation. It was created in the wake of recommendations by both the 9-11 Commission and the Iraq Weapons of Mass Destruction Commission that CIA focus more effort on gathering "open source" information — data that is available to anyone in the public, as opposed to information that can only be gathered through covert means.
Twitter used a veto clause in its contract with a separate firm to block the U.S. intelligence community from buying access to a Twitter data feed. The issue arose as early as September, when the private company Dataminr began work to transition a free pilot program for U.S. intelligence into a paying contract, according to a source in the technology industry. Dataminr is a New York-based firm founded in 2009 that has a contract with Twitter to purchase the raw feed of all Twitter data, known as the Twitter "firehose."
Dataminr has received investments from both Twitter, which reportedly owns a 5 percent stake in the company, and In-Q-Tel, the CIA's venture capital fund that exists to spur innovative technologies of interest to the intelligence community.
Among the many details of the saga that have not been made clear is exactly what information Dataminr contemplated selling to the CIA. A technology industry source, however, tells CNBC the data did not include direct messages, protected tweets, or geo-location information on where Twitter users are located, unless those users posted that information publicly. Tweets by their nature are public information — Twitter users broadcast their information to a wide array of followers, some of whom they know personally and some they do not. CNBC is among the news organizations that use Dataminr, which deploys algorithms to locate public tweets that may be an indication of breaking news anywhere in the world.
The Dataminr service has been particularly useful recently in spotting early tweets indicating that ISIS-related terrorist attacks were underway in Paris, Brussels, and San Bernardino, California, source said. Twitter's decision not to cooperate with U.S. intelligence is especially personal, given that ISIS threatened in a video released in February to kill Twitter's CEO Jack Dorsey and Facebook CEO Mark Zuckerberg.
A Twitter spokesman said: "We have never authorized a third party to sell data to an intelligence agency for surveillance purposes. Twitter data is largely public, and governments may review accounts on their own, like any user."
The spokesman also said that the company does not object to government use of Twitter data for nonsurveillance purposes. "For instance, if DHS wanted to buy Dataminr news alerts to surveil Black Lives Matter activists, we would decline. But if the DHS wanted to buy data to assist with disaster preparedness, would permit that."
A CIA spokesman declined to comment.
A U.S. intelligence official, however, told CNBC the government hopes Twitter will reconsider its decision. "This is a vital tool for situational awareness," the official said. "We're hopeful this is not the final decision and that there is some way to make this work. Why would they want the intelligence community to be at a situational intelligence disadvantage?"