That's an increase in net worth of $34.2 billion after deducting $1.8 billion of what he calls "economically meaningless" charges from Berkshire's purchase of minority interests in Marmon and Iscar.
That strong gain, however, was dwarfed by the S&P's 32.4 percent advance, including dividends.
Buffett noted that he expects to beat the benchmark S&P 500 stock index in years "when the market is down or moderately up."
"We have underperformed in ten of our 49 years, with all but one of our shortfalls occurring when the S&P gain exceeded 15%."
(Read more: Berkshire Hathaway's 15 biggest stock holdings)
Buffett does have one stat to celebrate. He points out that over the six-year "stock market cycle" of 2008 through 2013, Berkshire did beat the S&P, and "through full cycles in future years, we expect to do that again. If we fail to do so, we will not have earned our pay, After all, you could always own an index fund and be assured of S&P results."
Over the past five-year span, however, the S&P was the victor, the first time that's happened in more than four decades.
Berkshire did make a lot of money in 2013 with a record profit of $19.48 billion. That's an increase of 31 percent from last year's $14.82 billion.
Revenues totaled $182.15 billion, up from $162.46 billion.
More 'bolt-on' deals coming
Buffett told shareholders that while he and business partner Charlie Munger continue to "search for elephants, " large buys in the billions of dollars, many Berkshire subsidiaries are making smaller "bolt-on" acquisitions. There were 25 of them last year with a total value of $3.1 billion.
"Charlie and I encourage these deals. They deploy capital in activities that fit with our existing businesses and that will be managed by our corps of expert managers. The result is no more work for us and more earnings for you."
He expects "many more" in coming years. Their combined effect, he said, will be "meaningful."
Heinz 'partnership template'
Buffett said last year's purchase of a major interest in Heinz "created a partnership template that may be used by Berkshire in future acquisitions of size."
In that deal, Berkshire teamed up with 3G capital to buy Heinz. 3G is responsible for operating the company while Berkshire is providing much of the financing.
Buffett acknowledges some similarities to a private equity deal, but said there is a "crucial" difference. "Berkshire never intends to sell a share of the company."
Instead, he wrote, Berkshire would like to buy more and "that could happen!" Buffett said some 3G investors could sell to Berkshire or it may use some of its preferred shares to buy common stock.
(Read more: In letter, Buffett admits to $873 million mistake)
Operating results to date "are encouraging," but due to one-time charges and restructuring costs, Heinz contributed only "minor" earnings to Berkshire's record 2013 profit. Buffett, however, expects "substantial" earnings this year,
He boasted, "With Heinz, Berkshire now owns 8½ companies that, were they stand-alone businesses, would be in the Fortune 500. Only 491½ to go."
Todd and Ted
Buffett also did some boasting on behalf of portfolio managers Todd Combs and Ted Weschler, who both "handily" outperformed the S&P in 2013. They each run a portfolio valued at more than $7 billion and "they've earned it."
"I must again confess that their investments outperformed mine. (Charlie says I should add 'by a lot.') If such humiliating comparisons continue, I'll have no choice but to cease talking about them."
(Read more: Team Buffett stepping into the light)
Buffett added that the two also created "significant value" in "several matters unrelated" to their stock picking. "Their contributions are just beginning: Both men have Berkshire blood in their veins."
That kind of praise may contribute to speculation one of them, or possibly both, could eventually succeed Buffett as Berkshire's CEO.
Combs and Weschler— along with Buffett's third "T"— financial assistant Tracy Britt Cool, will be making arare public appearance Monday morning when they join him live at 8 a.m. ET on CNBC for a half-hour slice of "Squawk Box's " annual "Ask Warren" marathon.
'Fundamentals of investing'
Some of Monday's discussion will focus on a significant portion of Buffett's letter in which he shares some of his "fundamentals" of investing. (The section was published earlier this week by Fortune.)
He uses two of his own successful real estate deals to support a point he's made many times before: investors should buy stock as if they were buying a company and a share of its future profits, not betting on its price. They should, as a result, ignore the daily ups and downs of a stock's movements and keep their focus on the long run.
(Read more: Warren Buffett's three 'fundamentals of investing')
"Owners of stocks ... too often let the capricious and often irrational behavior of their fellow owners cause them to behave irrationally as well. Because there is so much chatter about markets, the economy, interest rates, price behavior of stocks, etc., some investors believe it is important to listen to pundits — and worse yet, important to consider acting upon their comments."
—By CNBC's Alex Crippen. Follow him on Twitter: