Warren Buffett is defending Goldman Sachs before shareholders at the Berkshire Hathaway annual meeting in Omaha, saying, "It's hard for me to get terribly sympathetic" with the alleged victim in the SEC's case against the Wall Street giant.

He began his marathon 5-hour (excluding lunch break) question-and-answer session with a lengthy explanation of the Goldman transaction at the center of the SEC's fraud allegations against the firm, using a Berkshire deal involving Lehman as an example.

While he concedes the SEC's accusations have "hurt" Goldman's reputation, he does not "hold against Goldman" the fact that it has been sued by the SEC.  The Justice Department has also opened a criminal inquiry into Goldman's trading.

• Warren Buffett to CNBC: Goldman Sachs Has "Lost the PR Battle At This Point"

Buffett says it shouldn't matter who is on the other side of a deal, and adds it's "hard for me to get terribly sympathetic" for ABN Amro, the firm identified by the SEC as the victim in the Goldman case.

CNBC/Lacy O'Toole CNBC's Becky Quick tapes an interview with Warren Buffett before today's Berkshire Hathaway shareholders meeting

Asked whether the controversy will affect Berkshire's investment in Goldman, he replies that ironically it is helping.  He thinks it has become less likely Goldman will completely pay off the loan, a move that would cut off the stream of large dividend payments Berkshire is getting.  "We love the investment."

GS loading...

Quote  |  Chart  |  News  |  Profile

[GS  Loading...      ()   ]

Berkshire loaned $5 billion to Goldman in September of 2008 and receives a dividend of 10 percent a year.  Berkshire also got warrants to buy Goldman shares at $115 each.  Their value fluctuates with Goldman's common stock price.  The loan dividends are not affected by the stock price.

Buffett does say he might change his mind about Goldman if the SEC charges "lead to something more serious."

He also expressed strong confidence in the leadership of Goldman CEO Lloyd Blankfein.  Asked who he would want to run the company other than Blankfein, Buffett replied, "If Lloyd had a twin brother, I would vote for him."

Appearing with Buffett, Charlie Munger says if he was on the SEC he would have voted against bringing the case against Goldman and also expressed confidence in Blankfein, although there are "plenty" of other corporate CEOs he's like to see replaced.

Neither Buffett nor Munger would have considered the Wells Notice Goldman received nine months ago as material, and thus the firm was not obligated to reveal publicly that it had received it.