"It's dead wrong to imply I've spoken to the board directly. Going to the board implies I've gone around Stumpf, the guy who is under fire," Buffett told CNBC. "I've talked to no one else on the board."
Earlier in the day, Douglas Kass of Seabreeze Partners Management wrote in The Street's Real Money blog that Buffett "expressed his extreme dissatisfaction" to the bank's board of directors and called for "a radical transformation of the bank's ethics, in no uncertain terms."
Kass later published another post on Real Money with the title "I Stand by My Post on Buffett and Wells Fargo." Kass has a small short position in Buffett's Berkshire Hathaway.
Buffett said that he spoke with Stumpf for about five minutes and told him the problem at the bank is bigger than the CEO may have originally thought. Buffett told the Wells Fargo CEO the fine "was not a metric to use in determining public reaction."
Stumpf told the House Financial Services Committee on Thursday that he has talked to "a lot of [Wells Fargo] investors" and that he has "had one coversation with Warren Buffet."
The billionaire also said that he did not think that Stumpf's interview with Jim Cramer on "Mad Money" went well. In the interview, the Wells Fargo CEO said the bank is sorry, but that he does not plan to resign.
Buffett's Berkshire Hathaway holds the largest stake in Wells Fargo with about 9.51 percent of shares outstanding, according to FactSet.
Wells Fargo was hit with $185 million in penalties for opening fee-generating accounts without authorization. Over a five-year period, the bank said, 5,300 Wells Fargo employees were fired over the practice.
In July, Buffett applied for approval to expand his stake in the bank.
— CNBC's Becky Quick, Alex Crippen and Dawn Giel contributed to this report.