Origin said its decision was also influenced by Thursday's news that Malaysian state-owned oil firm Petronas had agreed to pay $2.5 billion for a 40 percent stake in a liquefied natural gas plant planned by Australian energy firm Santos.
"It's not a surprise that Origin would turn it down. Santos' transaction yesterday has really set a new benchmark for the price of coal-seam gas," said Paul Johnston, a utilities analyst at Commsec Securities Ltd.
Santos said on Thursday the Petronas deal valued its proved and probable (2P) reserves at A$4.91 per gigajoule (GJ), or A$1.65 per GJ if using the largest estimate of the coal-seam gas reserves.
Analysts said that BG's improved deal valued Origin's 2P coal-seam gas reserves at about A$3.50 per GJ.
Origin said in a statement BG had increased its offer to A$15.50 per share in cash, valuing it at about A$13.6 billion and up from an initial offer last month of A$14.70 a share.
BG said separately it was surprised by Origin's decision and was considering its options.
Campbell McComb, investment director at Armytage Private, which holds Origin shares, said BG's raised offer was reasonable. "At A$15.50, in our eyes that's fair value or a bit better for Origin," McComb said. "The directors in rejecting it have set themselves up with a fair bit of work to do to justify how and why they rejected it."
Shares in Origin rose almost 9 percent to A$15.94 in the morning Sydney session, after rising to a record high of A$16.15. Its shares have surged around 80 percent since the start of the year.
Blow to BG
Analysts said Origin's rejection would be a blow to BG, which was hoping the deal would help secure the reserves it needs for a proposed Australian LNG plant fueled by coal-seam gas.
"Its quite difficult to find such large gas fields these days and there may be other energy majors eyeing Origin's assets. BG may decide to come back with a higher offer," said an analyst who asked not to be identified.
Origin said that a review showed that its largest estimate of coal-seam gas reserves, also known as proved, probable and possible (3P) reserves, had increased to 10,122 petajoules (PJ) from an earlier estimate of 4,578 PJ.
Its most likely recoverable reserves, also known as 2P reserves, jumped 91 percent to 4,715 petajoules.
BG said it thought the review was unrealistic.
Proposals by some Australian firms to use coal-seam gas as a feedstock for LNG plants have sparked strong interest in the country's coal-seam gas sector.
In a separate statement on Friday, Arrow Energy asked for a halt in the trading of its shares, pending an announcement on the sale of a stake in one of its projects.
Origin is Australia's only vertically integrated energy company with oil and gas production assets accounting for about a quarter of its revenues, which have helped to offset tighter margins in its traditional retail energy business.
Goldman Sachs and Gresham Partners are advising BG, while Macquarie is the adviser for Origin.