Now that the crisis has eased, that's a high price for Goldman to continue to pay for money it can borrow through more conventional channels.
The Journal says Goldman would need permission from the Federal Reserve to pay back its loan, but it isn't "clear" if Goldman has made a formal request for a go-ahead from the Fed.
But the paper says it has been told by "people familiar with the matter" that executives are "looking closely at whether to use a small chunk of the firm's $173 billion in excess liquidity to unwind the investment."
Under the 2008 deal, Goldman has the option of paying back the loan for $5.5 billion, depriving Berkshire of any future annual dividend payments.
That's why Buffett has joked that he tries not to answer the phone if he thinks Goldman is on the other end of the line.
As part of the loan deal, Berkshire also got warrants to buy $5 billion of Goldman common stock at $115 a share.
If they were exercised now, Berkshire would pocket roughly $2 billion in paper profits, by paying only $5 billion for stock currently worth just under $7 billion at the market price.
But Buffett has indicated that he doesn't plan to exercise those warrants until just before they expire in 2013, and could well hold onto them even if Goldman buys back the preferred shares it sold to Berkshire.
Current Berkshire stock prices:
For more Buffett Watch updates follow alexcrippen on Twitter.
Email comments to firstname.lastname@example.org