Margin of error: Why polling should always be taken with a pinch of salt

Key Points
  • U.K. polling companies have rethought their methods following two surprise votes in recent years
  • Pollsters say that inaccuracies are only called up if they clash with the overall story of an election
  • Polling based on observed data rather than direct questioning could be more accurate
A woman leaves a General Election polling station in the London Borough of Islington on May 7, 2015.
Rob Stothard | Getty Images

Supposedly incorrect polling has shaped politics in the U.K. and the U.S. in recent years. The improbable became the real following Donald Trump's election to U.S. President, the U.K.'s vote to leave the European Union last June and the Conservative party's victory in the 2015 General Election.

Pollsters have had to rethink their strategies. In March 2016, an independent review by the British Polling Council with regards to the last General Election revealed that pollsters had made two key errors: they had not contacted a sufficient number of traditional Conservative voters, nor had they appropriately accounted for the fact that younger demographics, though more likely to vote left, were less reliable at turning up on polling day.

Anthony Wells, director of political and social research at polling company YouGov, told CNBC via e-mail that the company had "invested in focusing (its) recruitment of those groups that were underrepresented in 2015: people less interested in politics, with lower levels of education or (who) did not vote in the last election." The accuracy of traditional telephone polling was also questioned. But Andrew Hawkins, chairman of pollster ComRes, argued that the method "will not become obsolete," offering that it was one of the few ways of guaranteeing a "truly random sample."

The Brexit referendum complicated matters. Hawkins said that Brexit was difficult to predict because it "motivated people to be politically active who wouldn't have been so otherwise." He argued that there was little evidence this sentiment would carry through to June's snap election, due, for example, to the U.K.'s first past the post voting system and the election being focused on more than one single issue.

UK election adds to the political noise: JP Morgan

The extent to which polling companies are called up for inaccuracy is reliant on the election story itself. Despite being very accurate in French presidential election's first round of voting, polls projected Emmanuel Macron's winning percentage in the runoff to be in the low 60s, though he ultimately scored 66 percent. Wells pointed out that this "didn't receive much notice" as it did not disrupt the election's overarching narrative. Hawkins said that close polling was more susceptible to error margins, and that inaccuracies are highlighted later on as "everyone loves a tight race."

For one expert CNBC spoke to, traditional polling across the board is challenged. Dave Elkington, CEO of sales acceleration platform InsideSales, explained to CNBC via telephone his view that "observed data tends to be accurate, volunteered data tends to be less so." Most polling falls within the latter category.

Elkington cited the example of the USC/Los Angeles Times Daybreak poll which accurately called last November's U.S. presidential election. This poll questioned the same group of voters each week, asking them to rank on a scale of 1 to 10 how likely they were to vote for each candidate. Rather than forcing potential voters into verbally committing to one candidate, this method intends to factor in people's "ambivalence."

Defending the polling industry against accusations of bias, Wells explained to CNBC that market research companies do not make money from political polling. Instead, the service was a "shop window" for companies to "advertise (their) accuracy." Deliberate bias would be "deeply damaging to our business" and "make no sense whatsoever," he said.

It remains to be seen whether the U.K. polling industry will accurately predict the outcome in June's General Election. Those in the industry that CNBC spoke to emphasized that the process was an art, rather than an exact science. "There's no magic pixie dust," Hawkins said.

Follow CNBC International on Twitter and Facebook.