The party is over for oil

When oil prices started taking off in 2004, it was part of the same "commodity supercycle" that sent prices of so many other commodities spiraling up. Oil stayed at the party longer than anyone else but now, the party is over.

The driving force was the surge in economic growth from the emerging markets — especially China. Between 2003 and 2013, China alone accounted for 45 percent of total growth in oil demand. It was a similar story for other commodities.

But the party ended earlier for other commodities as investment to expand production capacity coincided with a tempering of demand growth. Overcapacity and surplus, as opposed to shortage, became the dominating motif, as is so evident in the current travails of mining companies.

Prices for non-energy commodities had already begun to decline in the spring of 2011, according to the IHS Materials Price Index. More recently, that decline has turned into a rout, with prices down 50 percent just since July 2014, carrying them back down to about where they were in the depth of the 2008 financial panic.

Oil, however, had been the holdout, remaining at around $100 a barrel — and, in fact, reaching $115 in June 2014 when ISIS seemed to be rolling toward the gates of Baghdad. It was only in September 2014 that the weakening of oil prices became apparent. The decline turned into a collapse in November 2014, when OPEC made the historic decision not to cut output but, instead, let the market find its own price.

Oil price vs. non-energy commodity-price index

Source: IHS Materials Price Service

This difference demonstrates why oil is both "just another commodity" and a "geopolitical commodity." What kept oil prices up, even as the new U.S. shale oil production increased, was the roughly-commensurate loss of barrels owing to disruptions of one kind or another. In the failed state of Libya, oil production and exports largely stopped. At the same time, sanctions related to Iran's nuclear program took, at the peak, about 1.4 million barrels of Iranian exports off the market.

But the oil market ran out of offsets. U.S. production was continuing to surge, on track to add 1.5 million barrels per day just in 2014. At the same time, the risk of what IMF chief Christine Lagarde has called the "new mediocre" — lower growth in the global economy for a long time — meant that oil demand was growing more slowly — just 800,000 barrels per day in 2014. The "China chill" (slowing of the Chinese economy) was the single most important factor.

This was the context in which OPEC decided to hang up its jersey as market manager. Yet, to say "OPEC" is rather misleading. For this is not really a decision by OPEC, which is sharply divided. Rather it is the decision by Saudi Arabia, which holds most of the world's spare capacity, and the other Gulf countries to refrain from cutting output to keep up the price, as had previously been the practice for many years.

If they did cut their production, that would maintain the $100-a-barrel incentive for others to keep investing in new non-OPEC production. As Saudi oil minister Ali Naimi put it, why should they reduce their "low-cost" oil in order to end up subsidizing other countries' "high-cost" oil? That appears to be the Gulf countries' posture going into the OPEC meeting later this week.

How US drillers are beating OPEC's new oil order

Here again, oil is once more like "just another commodity," as producers of other commodities are struggling with the same issue — whether to cut production and reduce their market share to bolster price — or maintain output.

"Why should I make cuts" and make room for higher-cost rivals to step in?, asked a senior executive of one of the major mining companies. No longer does Saudi Arabia want, as Saudi Aramco chairman Khalid al-Falih said recently, to provide an insurance policy "free of charge" for higher-cost producers. In the face of the market fundamentals, he added, "The only thing to do now is to let the market do its job."

After the price collapse, the new U.S. shale-oil industry has proved to be more resilient than many had anticipated, as companies found that they could be much more efficient. In fact, U.S. output continued to increase through April. According to the IHS Oil Performance Evaluator, which tracks U.S. production down to the individual well, a dollar spent this month in the U.S. oil patch will be 65 percent more efficient than in 2014.

But efficiency is not enough to hold off the impact of lower oil prices. As companies struggle to cut budgets and, in some cases, to survive, U.S. oil production is now on a downward trend from that high point last April. By next April, we estimate it will be about a million barrels a day lower than April 2015. Around the world, projects are being postponed, drawn out, or simply canceled. All of this will show up in the future oil balance in years ahead.

Oil prices could rattle stocks in 2016: Bouroudjian

Oil's collapse does mark the end of the commodity supercycle. And that means that oil-exporting countries are now in the same boat as other commodity-exporting countries — struggling with yawning gaps in their budgets, low growth or recession, austerity and bitterness, and the potential for social and political turmoil. The countries that benefited so much from the heat of China's growing economy are now suffering from the impact of the "China chill."

This nexus of slowing China and deeply-troubled emerging market countries may end up an offset to a U.S. economy strengthening enough for the Federal Reserve to begin raising rates.

Meanwhile, oil's turmoil will continue. For new petroleum supplies are likely to come into the market in 2016. Assuming the nuclear sanctions are lifted on Iran in late winter or spring, Iran could bring what we currently estimate to be another 400,000 to 600,000 barrels of oil a day within several months. Iran's oil minister pegs the number higher — around a million barrels per day.

This specter of Iran battling to regain market share from its geopolitical rival Saudi Arabia and the other Gulf countries is likely to loom very large in the OPEC debates ahead.

Commentary by Daniel Yergin, the vice chairman of IHS and author of, "The Quest: Energy, Security, and the Remaking of the Modern World." Follow him on Twitter @danielyergin.