Read MoreWhat next for Khodorkovsky after his Russian release?
The ruling hits Russia at a time when it faces international sanctions about its role in Ukraine and anger over the downing of a Malaysian airliner over eastern Ukraine, where Moscow-backed rebels are fighting a separatist campaign.
The country is also grappling with slowing economic growth.
The Permanent Court of Arbitration in the Hague announced that Russia must pay the compensation to subsidiaries of Gibraltar-based Group Menatep, a company through which Khodorkovsky, once Russia's richest man, controlled Yukos.
Group Menatep now exists as holding company GML, and Khodorkovsky is no longer a shareholder in GML or Yukos.
Khodorkovsky, who is not a party to the action, was arrested at gunpoint in 2003 and convicted of theft and tax evasion in 2005. His company, once worth $40 billion, was broken up and nationalized, with most assets handed to Rosneft, a company run by Igor Sechin, an ally of President Vladimir Putin.
In a statement, Rosneft said it was "not a party to, and did not participate in the arbitrations, is not bound by the published rulings."
"Rosneft believes that all of its purchases of former Yukos assets, and all other actions taken by it with respect to Yukos were entirely lawful and proper applicable law," the statement said. "Rosneft does not believe that it may be brought against any claims in accordance with the published rulings or that these rulings can have an adverse effect on its business or assets."
Its shares were down 0.6 percent at 0830 GMT, while the RTS index of Russian shares was down 1.8 percent.
Separately, The European Court of Human Rights (ECHR) in Strasbourg is expected on Thursday to announce a decision on Yukos's multi-billion-dollar claim against Russia, ruling on 'just satisfaction' or compensation, a Yukos spokeswoman said.
Yukos's application in the ECHR, which is on behalf of all Yukos shareholders, argued that Yukos was unlawfully deprived of its possessions by the imposition of bogus taxes and a sham auction of its main asset.
In a case that Kremlin critics said offered a stark example of Putin's increasingly autocratic rule, Khodorkovsky was arrested at gunpoint in 2003 and convicted of theft and tax evasion in 2005. Putin justified the move by saying: "A thief must be in jail," quoting a popular Soviet blockbuster.
Read MoreKhodorkovsky isreleased into Russia dominated by Putin
Putin pardoned Khodorkovsky in December after he had spent 10 years in jail. He now lives in Switzerland.
The newspaper Kommersant, which earlier reported the Hague ruling, said the court ruled that Russia had infringed an international energy charter, adopted in 1991, that envisaged legal issues for investments in energy sectors.
The court also ruled, according to the newspaper, that Russia had to start paying the compensation by Jan. 2 next year, or face growing interest on the fine.
It cited GML director Osborne as saying GML will force Russia to pay out the compensation "if it wouldn't make payments within the court-defined timeframe".
Any funds won will be shared amongst the shareholders. The biggest ultimate beneficial owner is Russian-born Leonid Nevzlin, a business partner who had fled to Israel to avoid prosecution. He has a stake of around 70 percent.
A spokesperson for Nevzlin declined to comment.
The other four ultimate beneficial owners, each of whom owns an equal stake, are Platon Lebedev, Mikhail Brudno, Vladimir Dubov and Vasilly Shaknovski.
After he was jailed, Khodorkovsky ceded his controlling interest in Menatep, which owned 60 to 70 percent of Yukos, to Nevzlin.
GML shareholders are not expecting to claim twice, so if they receive monies pursuant to one case it would reduce their claim under the other, Osborne has previously told Reuters.