Why OPEC's plan to balance oil markets backfired

OPEC's plan to shake up the world oil market may have backfired for now.

Just a year ago, the Organization of the Petroleum Exporting Countries decided to let market forces determine the price of oil, rather than its own production quotas.

Conventional thinking then was that the U.S. oil patch would be littered with bankruptcies, and production would collapse. As for Russia, the world's largest energy producer would be forced to cut back production by hundreds of thousands of barrels this year due to both the weakness in oil prices and the impact of Western financial sanctions.

But the results have turned out very different. Instead of falling off, production increased from where it was last year, and the world is still swimming in oil. The three biggest producers — Russia, the United States and Saudi Arabia — have in fact been adding more than 1 million barrels a day more to the market in the past year.

"It really was a historic change. OPEC resigned and said it wasn't going to be the manager of the market. Let the market manage itself. The thought was that Russia would be hit harder than it's been hit," said Daniel Yergin, vice chairman of analysis firm IHS.

"The most frequent conspiracy theory was that the U.S. and Saudis conspired to bring down the price of oil to hurt Russia, which of course is ridiculous. The expectation was that U.S. production would collapse much more quickly than it has. Those were two of the big expectations of what would happen. I think the resilience of U.S. shale, even with the pressure on it now, has really been one of the biggest surprises of the industry."

The three major producers have not gone unscathed, including Saudi Arabia, which was the force behind OPEC's change in policy. The new conventional wisdom is that the price of oil will be "lower for longer," and will not show signs of a pickup until well into next year or later. As for OPEC, it meets again this week and it is expected to hold tight on its new pricing policy.

Read MoreThis is when oil will hit rock bottom: CNBC Survey

"I think the Gulf producers, led by Saudi Arabia, are expecting that this is going to be a perfunctory meeting, in the sense there's nothing new to do. There's no indication that they're going to abruptly change their policy. They are very concerned about market share. And they're very concerned about Iran coming back to the market, and how aggressive Iran is going to be in terms of trying to take markets back. The rhetoric about the oil market from Iran is pretty aggressive," Yergin said.

Read MoreCommentary: The party is over for oil

Iran, which has been economically crippled by export restrictions for years, hopes to start shipping oil in the first half of 2016 in the wake of this year's nuclear deal.

To deal with the new reality of persistently cheap oil, Saudi Arabia has been tapping the debt market, issuing bonds for the first time in seven years to help cover budget shortfalls. The Saudi Arabian state is a profligate spender.

Analysts say they expect Saudi Arabia to continue issuing debt, and the sense in the kingdom seems to be that it's better policy to issue debt when it can be done, than when it is really needed.

"They'll definitely be doing more bond issues in the next year," said Greg Priddy, director of global energy and natural resources at Eurasia Group. "Their level of government spending is predicated on oil prices around $100 a barrel. Prices where they are — that is not sustainable on a long-term basis." He added that Saudi borrowing levels — $27 billion expected so far — are fine now, but could be problematic if they continue over several years.

Saudi Arabia is an extremely low-cost oil producer, pumping crude at well below $10 a barrel. But its subsidy costs are high, and it has an elevated level of unemployment, particularly among the young. According to Stratfor, the kingdom is estimated to have $662 billion in reserves, and it spent about $84 billion between August 2014 and August 2015 to make up for the loss in oil revenue.

"It's our understanding that Saudi Arabia is involved in discussions to get large loans from Western banks. They have both access to credit and they have more in reserves. They're better positioned to ride out a longer storm while they tighten their belts," said Edward Morse, global head of commodities research at Citigroup. "The Russians don't have quite the same window of luxury as the Saudis. Over time, the Russian economy is in a more vulnerable state than the Saudi economy."

Morse said the Saudis have a more urgent need to find ways to employ locals. "They certainly have the money to do something. They should have done it a long time ago, by ending subsidies to the population and finding ways to diversify the economy," Morse said. He said the Saudis could have developed a petrochemical business or taken other steps to change the economic mix.

Read MoreThe party is over for oil: Dan Yergin

"They have the most vibrant stock market in all the Middle East. They are in significantly better position financially than Russia might be at the moment. Both would like to see higher oil prices," he said.

For the Saudis, the battle started with concerns about market share, and the rapid increase in U.S. crude production changed the global landscape over the last several years, reducing U.S. reliance on foreign oil, displacing barrels that once came to U.S. shores.

At the same time, the Saudis have been fighting to maintain market share in other parts of the world. For instance, Russia surpassed Saudi Arabia as the largest exporter to China this year, but Saudi Arabia used price cuts to overtake Russia for share and reclaimed the top spot in the most recent reports.

"They never themselves used the term of swing producer, but they did want to be the balancer of the market with spare capacity," said Yergin. Saudi Arabia has also gone after share in other ways.

"They are clearly building up their refining capacity to be an exporter of refined products, not just crude," said Yergin. "They have joint ventures in Asia but they have a big commitment to increase their refining capacity in Saudi Arabia."

The kingdom has also maintained its investment strategy, while others have cut back to the point where many major, early stage projects around the world have been shelved.

"They have the longest time horizon of any entity in the world oil industry. They have obviously been bringing down their foreign reserves, but they still have very ample reserves. It hasn't had the same effect you've seen in other places," said Yergin.

Russia, meanwhile, has surprised the market on several fronts, managing to keep production growing for now even with super-low prices.

"The Russian view is they will pump as much oil as they can all of the time and deal with the financial consequences after," said Chris Weafer, senior founding partner at Eurasia consulting firm Macro-Advisory. "They will never voluntarily cut production or reduce supply in order to manage oil prices. That is simply not going to happen. It's out of the question. ... Occasionally you get speculation Russia is talking to OPEC. That's cosmetic. It's never going to happen."

As in the U.S., Russia's oil producers have improved efficiency by using new technologies to improve production for some of their old wells. But Russia has been hampered in developing new Arctic, deep-water or shale projects because of the financial sanctions placed on it by the West after it invaded Ukraine.

"This year, oil is up 1.5 percent despite the sanctions. What we hear is the companies are having to deal with the fallout from sanctions, but it's not the lack of access to engineering services or spare parts," said Weafer. "The big problem they all have to deal with is the financial sanctions ... oil companies are having to pay down external debt, and they're not able to replace that." External debt was $740 billion in January 2014, and it's now around $500 billion, he said.

"They've been forced to deleverage across the board," Weafer said. But because the companies are becoming more efficient, he does not anticipate a big drop in oil production.

"Technology and efficiency have been (big factors), and there's no reason to expect there will be a big falloff next year. If sanctions stay in place for another 12 months, you wouldn't expect Russian oil production to change too much. There's speculation there'd be a big drop down of 500,000, 700,000 (barrels a day). That's still not in the cards. At worst, it could be a slippage of a couple hundred thousand. The efficiency gains have kicked in."

But another factor has kicked in as well, and that has proven the key for Russian production. The Russian central bank's policy of letting the ruble float with oil prices has helped keep costs down.

"Russians have said they can't cut. Oil and natural gas has provided about half of their budget but they've been insulated ironically by the fall in the value of the ruble," said Yergin. "The dollar price of oil may be down. So is the value of the ruble. So domestically, it hasn't crimped their spending as much. This has been very good for their service industry. What has been circumscribed is the role of Western companies, but that's not because of the fall in oil prices. That's because of the sanctions."

Weafer said that in 2013, Russia needed international bellwether Brent crude to run at $113 a barrel in order to balance its budget. That has changed. "This year the budget will balance at $73/74. There's been a big reduction in the oil price needed to balance the budget."

Russia's attempts to cut its debts due to the sanctions are likely to leave its industry in a stronger position to acquire new debt, new investment and ultimately, to grow, Weafer said. "I don't mean to imply that everything is fine here. There's consequences of the weak ruble, which has been positive for the budget deficit and cutting imports. The bad side is it caused inflation to ratchet up ... and it means the central bank had to move up its benchmark rate to 17 percent last December." The interest rate was back down to 11 percent in the summer.

"This has not come without considerable pain. The difference is people have not reacted to it. Russian companies tend not to cut jobs. They cut salaries rather than jobs," Weafer said.

But Morse said Russian companies are expected to face new taxes, and that could challenge their costs. Russian companies though have an average production cost below $20 a barrel.

Russia's success with letting its currency devalue seems to be in part behind the speculation that Saudi Arabia and other Gulf producers could seek to de-peg their currencies from the U.S. dollar. But analysts do not see a high likelihood of that for now.

The Kremlin, meanwhile, has its own problems with sanctions, and Weafer said it's a priority to have them removed. "The oil price collapse is what brought Russia into recession, but oil price recovery will not bring Russia back to growth," he said, noting that the removal of sanctions on its financial sector "is critical to return the country to growth."

Unlike other producing countries, the U.S. has no state ownership of oil companies, and the industry is a collection of hundreds of companies large and small that can produce oil at an estimated cost of $30 to $60 per barrel. According to Yergin in 2014, 80 percent of the new production in the U.S. in 2014 came from 30 percent of the new wells.

Read MoreHow US drillers are weathering OPEC's new order

Citigroup estimates that production costs for shale producers will fall by 25 to 30 percent through 2016 and offshore costs will fall slightly less. But even with cheaper costs and greater efficiency, the industry has been feeling the pinch.

There have been bankruptcies, and indebted companies are expected to increasingly have trouble getting financing. Citigroup sees capex spending for new projects dropping 48 percent in 2016 through 2020, from the preceding five years. It expects offshore to be more impacted than other production.

"I think the Saudis, among others, have been surprised by the robustness of U.S. production and the sticking power. Finally, we're seeing clear signs of a rollover in production. We don't know the degree to which it is rolled," said Morse. "It's clearly being impacted."

He said U.S. deep-water production is not being affected, but one vulnerable area is stripper well production, or small wells that produce 15 barrels a day or less. "They are mostly ma and pa rather than big companies. It's likely they'd suffer from negative cash flow because they don't want to lose their right to pump," he said. They could result in losses of 400,000 barrels per day in production next year, and about 300,000 this year, according to Morse.

"It's reasonable to think (U.S. production) is about 9.2 million (barrels a day) now," he said. "We think it'll fall depending on what the rig count is into the summer of next year, maybe down to around 8.7 million."

The worst could be ahead for the U.S. industry if oil prices stay low. For one, financing could become a problem, particularly for companies with high-yield or "junk"-rated debt. Analysts say it's becoming clear that some companies did not hedge in the derivatives market against such a steep and long-lasting downturn in oil prices.

"Hedges roll off in Q1. When they get to Q2, they'll have more naked exposure to the oil price. We expect more consolidation in the industry in the U.S. Whether or not it's looked at as a reduction in production, it will be looked at as bullish by the financial markets. We think the combination of the high-yield universe of companies seeing their hedges roll over by Q2 and certainly the redetermination of reserved-based lending in April, producing more of a hit on the financial stability of these companies," Morse said.

As for the future, in a world where prices ultimately stabilize, the big three producers are all seen playing a large role, but perhaps somewhat differently.

Morse expects to see the U.S. government ultimately end its self-imposed ban on oil exports, and that will make the U.S. a bigger force in driving global oil prices.

"I think the U.S. combined with the spread of shale is going to be a phenomenal factor in boosting market principles going forward. The Russians will be trying to use oil as an instrument of foreign policy in their neighborhood, and China is one such neighbor. It's not clear to me, at the end of the day, that Russia has more leverage," he said.

"The Saudis are handicapped by their proclivity to use energy as an instrument of foreign policy in a world where the buyers have more leverage over them," Morse said.

"I don't know how to think about how the Saudis try to implement their national interest through oil. They will almost need to buy a market share by building refineries around the world like they've done in the U.S., and like they are trying to do in China," he said.