Oil firms are swimming in data they don't use

McKinsey & Company wanted to know how much of the data gathered by sensors on offshore oil rigs is used in decision-making by the energy industry. The answer, it turns out, is not much at all.

Susana Gonzalez | Bloomberg | Getty Images

After studying sensors on rigs around the world, the management consulting firm found that less than 1 percent of the information gathered from about 30,000 separate data points was being made available to the people in the industry who make decisions.

Technology that can deliver data on virtually every aspect of drilling, production and rig maintenance has spread throughout the industry. But the capability—or, in some cases, the desire—to process that data has spread nowhere near as quickly. As a result, drillers are almost certainly operating below peak performance—leaving money on the table, experts said.

Drilling more efficiently could also help companies achieve the holy grail—reducing the break-even cost of producing a barrel of oil, said Kirk Coburn, founder and managing director at Surge Ventures, a Houston-based energy technology investment firm.

Separately, a report by global business consulting firm Bain & Co. estimated that better data analysis could help oil and gas companies boost production by 6 to 8 percent. The use of so-called analytics has become commonplace in other industries from banking and airlines to telecommunications and manufacturing, but energy firms continue to lag.

Read More

The implications are real, according to McKinsey. In studying the offshore sector, the firm found that rigs in the North Sea, its largest sample group, were up and running as planned only 82 percent of the time. Improved use of data could result in better up-time.

"Most oil companies have a target of 95 percent. That's a huge gap, and a lot of lost production based on unplanned outages and maintenance," Tor Jakob Ramsoy, head of McKinsey's Norway office, told CNBC.

The problem is that while oilfield sensors offer real-time data on operations, the information is usually used to make immediate, binary decisions—either do this, or do that—rather than being stored, filtered and analyzed to inform future decision-making.

Exxon trims capex to average less than $34B
Exxon trims capex to average less than $34B
Exxon's global risk factor
Exxon's global risk factor
February job cuts top 50,000: Challenger
February job cuts top 50,000: Challenger

To be sure, it's important to differentiate between short- and long-term planning, said Satyam Priyadarshy, chief data scientist for Halliburton's Landmark software division. Most data are used for future planning, but that might extend only 15 minutes into the future, such as during drilling. In that sense, energy producers are good at using the vast majority of their data.

The biggest challenge the industry faces is so-called "data democratization," or making the data available to decision-makers across several departments in a way that allows everyone to use it meaningfully, he said.

"If data is in silos, these are all closed-shell systems," Priyadarshy said. "When talking about exploration and production as a whole of the universe, you want to have data that is easily accessible."

Read More Expect low prices, more volatility in oil: Exxon CEO

That's all the more challenging because of the organizational structure of the typical oil driller, which can prevent data from flowing freely among decision-makers, said Riccardo Bertocco, a partner at Bain & Co.'s Dallas office.

He explained that information coming from seismic studies feeds into a reservoir model run by the subsurface staff, while data from the drilling and well completion phases go through the wells group. Meanwhile, the operational production group handles the information on how hydrocarbons are flowing through the well.

"What happens in organizational structure, those three groups are separated," he said. "You can't really make integrated field decisions unless you have all pieces of the information."

The sleepy town sitting on $2.5B worth of oil
The sleepy town sitting on $2.5B worth of oil

Integrating that operating model is just one piece of the puzzle. Petroleum and drilling engineers do not typically have the skills needed to use advanced analytics in order to process data. At the same time, data scientists are seldom trained in petroleum engineering, geology or other specializations of the oil industry.

For that reason, Bertocco said, drillers must integrate teams of oil engineers and professionals with data specialists, as well as information technology staff who can manage the third and final piece: a technology platform that allows all decision-makers to access information.

Read MoreIraq, Iran and Libya are wildcards for oil: Expert

While exploration and production companies may lack the ability to process "big data" (click here for an explanation of what that means) the Silicon Valley firms that provide data infrastructure do not always understand how to apply advanced analytics to the oil industry, said Sashi Gunturu, founder and CEO of oilfield analytics firm Petrabytes.

"These are all scientific problems, so you need a good workflow rather than an app or a product line," Gunturu told CNBC.

He noted that the issues drillers face align with the "three Vs" of big data: volume, variety and velocity.

Prior to drilling, companies are dealing with large volumes of data as they perform seismic studies on many potential well sites. During drilling, they are taking in a wider variety of data. Once the oil is pumping, the velocity of information becomes an issue as producers must process reams of constantly streaming data.

Read MoreSome tech firms winning while oil is falling

Few drillers are analyzing huge volumes of data yet, but many are piloting advanced analytics programs. The future benefits could be significant, especially for the companies that operate in high-cost basins, such as North American shale and tar sands producers.

If energy firms can make new drilling much more efficient, they could achieve the holy grail—reducing the break-even price of a barrel of oil, said Surge Ventures' Coburn.

Coburn believes a lasting shift toward greater technology use in the oil field will happen when younger executives take the reins, but he said drillers may expedite tech purchases now that oil prices are about 50 percent below their June highs.

"When oil prices were high, it was a nice-to-have," he said. "Now it's becoming a have-to-have."