What is the best predictor that people will stay at their jobs? That friends work there. Are felons undesirable employees? For certain jobs, no, and at call centers they tend to perform better than those with a clean record. These are the sorts of correlations that big data—the ability to crunch mammoth amounts of information to uncover fresh insights—forces on business.
The cold, hard data fly in the face of subconscious biases and challenge longstanding practices that were hit-or-miss. Big data is shaking up all aspects of society. It will make companies more efficient and employees more productive, as Viktor Mayer-Schönberger of Oxford University and I document in "Big Data: A Revolution That Will Transform How We Live, Work, and Think."
In some ways, algorithmic HR is not entirely new. It continues the tradition of "scientific management" begun a century ago when Frederick Winslow Taylor shadowed workers with a stopwatch. And companies have long used software to screen the tsunami of résumés they receive to help assess how a candidate's background matches the needs of the job.
What is new is the huge scale of data available and that they work better than they did. Big data doesn't do anything particularly different for businesses than what they have always tried to do: Hire the right workers and get the most from them. But there is a good chance that they can do this much more effectively.
(Read more: Big Data Becomes a Market Darling)
In that respect, applying big data to managing a workforce should be a welcome development. It will test and perhaps overturn many sacrosanct assumptions of the labour force that are overdue for rigorous examination.
For instance, many companies looking to hire hourly workers eliminate candidates who are deemed job-hoppers. But this is badly short-sighted: There is no evidence that their performance is any different than that of employees who stick around longer, according to a recent study of 100,000 employees by Evolv, a big data HR agency.
Yet as they adopt algorithmic HR, companies must stay vigilant. Big data can unlock new efficiencies, but it can also be as misleading as the knee-jerk decisions and untested presumptions of the personnel manager of yore. One risk is simply statistical—that of spurious correlations. When the number of variables expands dramatically, the possibility of discovering chance correlations grows.
Another risk is biases in the data set itself. A finding that women don't cut it as stock traders says less about their capabilities and more about the macho culture of the trading pit that results in fewer women persevering there. Relying on the data and hiring only men would not only perpetuate the situation but deprive firms of potentially excellent female traders.
(Read more: Bank Data Cashes in on Your Feelings)
Even more troubling, innocent findings may be a proxy for something noxious. For example, a good predictor that employees will stay at their jobs is the distance they have to travel between home and work (the shorter the better). But using this information is tricky: It may inadvertently enshrine race as a basis for hiring.
In our book, we call for creating a professional class of experts, "algorithmists," trained in big data analysis who can scrutinize a system to ensure that it adheres to best practices and does not result in unfair outcomes. Big data cannot be a black box. (If nothing else, algorithmic HR will likely represent the Full-Employment Act, for not only statisticians but also lawyers.)
The trickier problem is what to do when the correlations are robust but unfair. In such cases, we need not genuflect before big data but shape its implementation according to our values. One model to consider is how the European Union treated insurance premiums that discriminated by sex. Because men get into more accidents, they paid more. This wasn't sexism but statistics. Yet the EU judged it unlawful. As of last December, insurance firms were required to be blind to gender.
Similarly, businesses can decide how they wish to act on the findings of big data. And regulators can step in if something is an affront to basic fairness.
(Read more: Data Premium Rises in Health Care)
Considering that companies spend a fortune building a workforce yet so many new hires don't work out, using data to improve how businesses hire, retain and motivate employees can result in an outsized gain in corporate performance. At a time when chronic unemployment bedevils the West, it may even mean more jobs. But it requires that we do not fear our algorithmic overlords.
Kenneth Cukier, the data editor of The Economist, is the co-author, with Viktor Mayer-Schönberger, of "Big Data: A Revolution That Will Transform How We Live, Work, and Think" (Houghton Mifflin Harcourt, 2013).
Cukier is the co-chair of The Economist's upcoming Information event.