The company's shares fell $8.08 as of 11 a.m. ET Tuesday after a weak earnings report, theoretically costing the Oracle of Omaha about $640 million.
He is also IBM's largest shareholder, with almost 20 million more shares than the next-biggest investor.
To be sure, Buffett takes a very long view, and he added to his IBM holdings earlier this year. As recently as May, he said he expected to make "considerable" money on IBM over the long term.
But it's also not the first time IBM has cost Berkshire a lot of money on a bad quarter.
Last October, IBM's sharp drop after earnings wiped just over $1 billion off Berkshire's books. This past July, another IBM weak quarter cost him more than $700 million the next day.
IBM said Monday it had beaten analysts' earnings estimates, but missed revenue expectations for its third quarter.
The tech giant reported earnings of $3.34 per share on $19.28 billion in revenue. Analysts had expected IBM to report earnings of about $3.30 a share on $19.62 billion in revenue, according to a consensus estimate from Thomson Reuters.
"EPS is affected by the value of the dollars you earn overseas which just aren't as strong," Ariel Investments' Charlie Bobrinskoy told CNBC. "We've got weakness in Asia, we've got some weakness in parts of Europe. So I think IBM's a perfect example. We're very long term bullish on it but I think it's going to be a very choppy quarter."
—CNBC's Everett Rosenfeld contributed to this report