The biggest energy dog bet of all, or value trap?

Some contrarians have been singing the praises of an out-of-favor energy investment. Nope, not oil and gas.

Love it or hate it, coal is still pretty hard to ignore for investors who look across the market's breadth, even amid the current sector bloodbath. Simply put, it's big, still used to generate an estimated 40 percent of electricity needs in the United States and tethered to what's expected to be the biggest growth story of all in the world for decades to come: China.

Jim Urquhart | Reuters

So is coal the ultimate energy value play, or is it a value trap that opportunistic stock pickers should stay far away from because it's never coming back?

The coal sector's investment outlook is a paradox.

What's not to like? Plenty.

Coal stocks have been clobbered, and there remain few signs of life in the sector. The Dow Jones Coal Index has plunged 36.5 percent in the past year, and in the past five years (brace yourself) it is down almost 75 percent—87 percent since before the last recession began in 2007.

Read MoreStay focused on stock market in 2015: Advisors

Of coal companies that have publicly traded debt, Moody's Investors Service and Standard & Poor's rates all their bonds as junk. "If you look at the long term, it's not getting any better," said Standard & Poor's analyst Aneesh Prabhu. "It's a secular decline."

Almost none of the companies are currently growing sales—a recipe for stock inertia, at best—and at a time when they're beset by utilities' switch to natural gas, prompted by hydraulic fracking, a transition that skeptics still don't think the sector can manage. Natural gas prices collapsed beginning in 2008 as new supplies hit the market, changing probably forever the competitive landscape for fuels used in the electricity business.

In both the U.S. and Europe, regulatory changes now in the pipeline are expected to make new coal-powered electricity plants economically unfeasible, and even accelerate the mothballing of existing coal plants.

"We have the absolute destruction of the coal industry. If you think it's coming back, you don't understand the business ... because it's not going to come back." -Bob Murray, CEO of privately held Murray Energy

What's to like?

It's been a political football in Washington, D.C., for years, but now has Republicans, emboldened by midterm elections, planning a push to roll back President Barack Obama's proposed limits on electricity plant carbon emissions. That kind of Capitol game can often create short-term trading windows.

In the U.S., electricity plants account for about 85 percent of coal demand, with higher-priced coal destined for steelmaking plants making up most of the rest, said Moody's analyst Anna Zubets-Anderson.

Moody's thinks new EPA rules could cut U.S. coal demand 20 percent by 2020, cutting into the 39 percent share of the market for electricity fuel it held last year, with up to a quarter of coal-fired electric plants closing. Standard & Poor's thinks coal will produce only 27 percent of America's electricity by 2025.

It's not as if the industry hasn't noticed. Bob Murray, CEO of Murray Energy, the largest underground coal mining company in the U.S. and an outspoken critic of regulation in general and the Obama administration in particular, raised eyebrows with a September energy conference speech in which he cited U.S. Chamber of Commerce data that coal might supply only 14 percent of U.S. electricity fuel by 2030.

"We have the absolute destruction of the coal industry," said Murray, whose company is privately held. "If you think it's coming back, you don't understand the business ... because it's not going to come back." Murray predicted that prices would not recover until 2016, if then, and several coal companies would be bankrupt soon. He declined to elaborate on his September comments.

Is it a war on coal jobs?
The coal industry's argument has always been that it is good for the country because it generates lots of jobs and creates cheaper electricity, an argument Murray repeated when suing the EPA over the proposed carbon standards this summer.

Coal mining is down to just 75,900 U.S. jobs, from 175,000 in 1985 and about 80,000 during 2009, when President Barack Obama took office, according to the Bureau of Labor Statistics. But at current rates, the U.S. economy would replace those jobs in a week if they all disappeared tomorrow. In Kentucky, where Sen. GOP Leader Mitch McConnell used the administration's "war on coal" as a rallying point in this year's election, the 11,000 coal-mining jobs equal the number created by the state's economy in October and represent less than 0.6 percent of Kentucky jobs.

New regulations have less to do with coal's job losses so far than the explosion of natural gas does, Zubets-Anderson said. One piece of evidence: Overall, mining and extraction employment has risen by about 5,000 jobs in West Virginia, the nation's second-largest coal producer behind Wyoming, to 33,200 since 2009 as hiring by oil and gas drillers has offset cuts by coal miners. Wyoming has only 6,400 coal mining workers, down 500 since 2008.

The regulations will effectively squeeze out any rebound in U.S. utilities' demand for coal, Prabhu said. Most U.S. coal plants are at least 35 years old, and the capital investment needed to upgrade them to comply with the Obama administration's proposed carbon limits would require running many of those until they are 65 years old or more, well past the average useful life of about five decades, Standard & Poor's Prabhu said.

Investors should not be fooled into thinking it is just regulation that has crimped coal demand and all it would take is a pushback against the EPA and Obama administration to change the equation. Utilities are especially wary of coal with natural gas trading at less than $4 per million BTUs, and in the range of $5 or lower over the past five years, except for a brief spike above that last winter.

"A lot of utilities have lost their appetite for coal,'' Prabhu said. "If you don't think gas is going to $6, you won't make the investment. It's throwing good money after bad."

Even as Republicans begin efforts to roll back some carbon measures, none of the coal companies are banking on a major rebound in U.S. coal usage for electricity. They are looking elsewhere.

Pivoting for profits

The sector outlook may get a little better in the medium term, as growth in Asia gives companies like Arch Coal, Peabody Energy and Cloud Peak Energy a chance to pursue new markets, even if confidence in that opportunity is wobbly right now. The problem with this global pivot is that coal companies have been talking it up for a few years already, but just as demand in China began to slow.

Coal exports to China are down almost 85 percent since early 2013, according to the U.S. Energy Information Administration. And Australia, where Peabody also has operations, has emerged as a key competitor for China's imported-coal business.

International prices for metallurgical coal used to make steel have plummeted by more than half since 2011, according to the EIA, hurt by a growth slowdown in China that has cut into commercial construction in the world's most populous nation.

Read MoreCNBC Explains: The energy demand forecast

The International Energy Agency predicts that global demand for coal will grow at an average annual rate of 2.1 percent per year through 2019.

"China will continue to depend on coal for economic development and affordable energy," Peabody Energy CEO Gregory Boyce said. "We're confident coal will be [China's] dominant energy source for decades.''

Consol Energy, meanwhile, has diversified and now gets most of its revenue from natural gas—it recently released plans to spin off some coal operations into separate entities. Nacco is a mini conglomerate that also sells small appliances. Closely held Murray is buying up other mines—it bought half of Consol's coal operations in 2013—to drive down unit costs.

"I don't know what will happen in the long run, but in the short term Consol's strategy is working best," Zubets-Anderson said.

That's obvious from the stock performance, down 15 percent in the past one year versus "true" coal stocks like Arch, down 65 percent.

The third leg of the rebound strategy, cost-cutting, has gotten the most traction in the opinion of Stifel Nicolaus analyst Paul Forward.

Coal company operating-expense trend

Arch Coal

Quarter ended 9/30/2013: $1 billion
Quarter ended 9/30/2014: $777 million

Cloud Peak Energy
Quarter ended 9/30/2013: $341 million
Quarter ended 9/30/2014: $256 million

Consol Energy
Quarter ended 9/30/2013: $750 million
Quarter ended 9/30/2014: $832 million

Peabody Energy
Quarter ended 9/30/2013: $1.7 billion
Quarter ended 9/30/2014: $1.7 billion

(Source: Google Finance)

A coal business can still be marked by pretty solid profit margins, at least before interest, taxes and noncash expenses. Cloud Peak made $130 million of EBITDA on $982 million in sales in the first nine months of 2014.

But the industry's average debt load is eight times EBITDA, bloated by leveraged buyouts and other deals. Coal companies have significant debt-to-total asset ratios due to the M&A craze that came right before the pace of the secular decline quickened and natural gas prices crashed.

Coal company debt-to-total assets ratio

Arch Coal: 58 percent
Cloud Peak Energy: 25 percent
Consol Energy: 28 percent
Peabody Energy: 44 percent

(Source: Google Finance, quarter ended 9/30/2014)

Even well-regarded operators, like Cloud Peak, with relatively low levels of debt compared to some peers, are shrinking, with the company's sales down 5.8 percent this year. Cloud Peak shares are down much more, with a 50 percent decline in the past year.

Read MoreCNBC Explains: EBITDA

"Some companies still have potential as investments, coming as they are off a very depressed base, Forward said.

"The risks are well understood: weak Asian import markets for steam and metallurgical coal; political headwinds and increasing competition for coal in the U.S.," Forward wrote recently. "The potential rewards tend to be realized when oversupplied coal markets shift to undersupply. We anticipate some emergence of improved utility coal pricing in 2015–2016."

The question is how long such a pickup could last, Prabhu said. As much opposition as the coal industry has drawn from environmentalists, the skepticism of markets may be even more dangerous to its future.

As they pivot to new markets that aren't growing as expected, and new business models that have to cope with big regulatory burdens, coal company profits are way down and their prospects are uncertain. But some contrarian investors are still betting on value in the sector.

Investor (and CNBC regular) Dennis Gartman had this to say of buying up coal companies this past August: "It is a long-term leap on continued economic growth. Buy four or five of them. Two of them probably will go out of business. Two of them will probably double or triple, and one of them may be a 10-bagger after all the smoke clears."

The smoke hasn't cleared yet. Since Gartman made his call, the coal sector has continued on a path that investors targeting the sector have gotten used to—down, if not out.

Coal company sales trend

Arch Coal
Quarter ended 9/30/2013: $791 million
Quarter ended 9/30/2014: $742 million

Cloud Peak Energy
Quarter ended 9/30/2013: $375 million
Quarter ended 9/30/2014: $342 million

Consol Energy
Quarter ended 9/30/2013: $783 million
Quarter ended 9/30/2014: $877 million

Peabody Energy
Quarter ended 9/30/2013: $1.8 billion
Quarter ended 9/30/2014: $1.7 billion

(Source: Google Finance)