When a Rasmussen poll last month showed Representative Roy Blunt opening a double-digit lead over Robin Carnahan in their Senate campaign in Missouri, John Hancock was not surprised.
Kick Images | iStock Exclusive | Getty Images
Mr. Hancock, a political consultant advising the Blunt campaign, had seen a similar shift in public opinion days earlier, through a software tool that analyzed the language being used in conversations about the campaign on social networking sites, blogs and other online conversations.
He said this technique, known as sentiment analysis, would soon be a part of every campaign he works on, because it helps him determine quickly which messages are resonating with potential voters. “You get a real sense of who’s carrying the day,” he said. “It affects the advice you’re able to give.”
Online organizing techniques have been rapidly adopted by the political world, and they played an important role in President Obama’s victory in 2008. Now, campaigns and the news media are becoming convinced that the Internet can also be mined systematically for useful data about public opinion. The New York Times has a tool that monitors Twitter for posts about candidates.
Businesses were quick to embrace sentiment analysis to monitor the performance of their brands. The programs scrape online messages for mention of certain products, then determine whether the language is positive or negative.
While political hands say there is interest in adapting these techniques to campaigns, the pace is more cautious. The analytical rigor can vary widely, they say, and there is uncertainty about how to use these tools. But Mr. Hancock and others say that such techniques could become common by 2012.
Technology companies that do sentiment analysis say their tools are already providing useful results. Linguamatics, a British company, analyzed posts from more than 130,000 Twitter accounts to gauge public opinion during the British elections this spring. The company’s analysis yielded similar results to traditional political polling, and predicted within one point the percentage of votes the Conservative Party would win.
Crimson Hexagon, a technology company in Massachusetts, analyzed expressions of public sentiment across the country about the oil spill in the Gulf of Mexico. Its analysis showed that people who lived near the gulf had a lower tendency to assign blame, focusing instead on the logistics of the relief efforts.
The ability to provide data on public opinion in real time is a primary attraction of sentiment analysis. Because the technique passively monitors conversations, it can track which ideas develop organically, something that is unlikely to happen in traditional polling when respondents reply to specific questions.
Changes in the way people communicate — particularly the increasing number of people who do not have landline telephones — also present challenges for traditional polling. The Pew Research Center released a report in October finding that polls that do not use cellphones in their samples can overestimate support for Republican candidates by four to six percentage points. Some advocates for sentiment analysis see this as evidence of a need for new techniques.
“We’re not necessarily seeking to replace — immediately, in 2012 — the traditional mechanism. But it’s got to have a seat at the table,” said Michael Urban, who worked on several Republican campaigns and on polling for Mr. Hancock’s political consultancy, before starting Globalpoint, a start-up that develops sentiment analysis tools for use in politics.
To be sure, there can be pitfalls in divining public opinion from online musings. The people who post their political views online are not a representative sample of the population, either demographically or in their level of political engagement. Further, the messages often come with little or no information about the person who posted them.
Your Money Your Vote - A CNBC Special Report
Sentiment analysis tools also have a tin ear for sarcasm, and are easily distracted by active but irrelevant conversation. During the recent British elections, Linguamatics recognized a huge spike in positive language about David Cameron, the Conservative Party candidate for prime minister, in the first moments of a televised debate. It was almost entirely because of a message on Twitter that had been resent many times that jokingly claimed that a television network had already declared Mr. Cameron the winner of the debate.
But the lack of scientific precision does not mean that this technique is not useful, say those familiar with it. Sentiment analysis just has to find its role.
On election night on Tuesday, CNN plans to use sentiment analysis to identify and track themes that gain traction online. These could be opinions on the political parties — general support for the Tea Party movement, say — or on the issues.
But David Bohrman, CNN’s Washington bureau chief, says that the most attractive part about the system has been its ability to monitor the conversations that are taking place independently, rather than having to frame the issues by asking direct questions.
“We’re waiting to see what they’re going to say,” he said.