"I thought, 'This sounds too good to be true,'" says Buffett, speaking with CNBC. "I am a suspicious guy by nature, so she didn't sell me."
But then Buffett's wife took him to visit the congregation and meet its leader.
"It is run by a very special man, Cecil Williams," Buffett says. "She took me there one time to Glide and I got to meet Cecil, I got to see what happened, the kind of people he was helping and how he was helping them."
The Glide church is located in the Tenderloin district of San Francisco, an area known for its crime, homelessness and drugs.
Williams was assigned to be the minister of the Glide church in the early 1960s. At that time, the church was "on the way down," says Buffett.
"It had about 110 or 20 white parishioners and Cecil was black, so the 120 went down very fast when he arrived there. And so he started from scratch and he built a remarkable church — but really a remarkable social organization."
Indeed, Glide, under the leadership of Williams, grew into an organization with an impressive social justice program. Glide serves three squares a day to the homeless and the hungry. In its last fiscal year running from 2015 to 2016, Glide provided more than 770,000 meals. Glide also offers affordable health care to the homeless and poor, including mental health care and non-Western holistic treatments. Glide has programs to help those who are dealing with substance abuse, for the testing and prevention of HIV and AIDS and for the treatment of diabetes. Glide offers free legal services too, and those are just a few of the many programs.
"He helps the people that the world has given up on and he never gives up on anyone," says Buffett of Williams.
"They really take people who have hit bottom and help bring 'em back. And he has been doing it for decades, and he's a remarkable man and if we can help out by raising some money for him, I enjoy doing it."
To raise money for the Glide church, Buffett has auctioned off his company over lunch for 18 years running. This year, the lunch date, which happened last week, went for $2.7 million to an anonymous bidder. Buffet's yearly auctions have raised $25 million for the church.
"The generosity of Warren Buffett and the eBay for Charity auction bidders allows us to build bridges of compassion, hope and love for everyone, regardless of age, race, gender or country of origin," says Williams in a written statement.
"Mr. Buffett is a steadfast believer in the power of inclusion and opportunity, and that's what we provide to San Francisco's most marginalized residents. Through our programs, we are a beacon of hope and a sanctuary of unconditional love for all."
Lunch with the Oracle of Omaha has gone for as much as $3.5 million, in 2016 and 2012. The lunch wasn't always so pricey. In 2001, the second year it was auctioned off, the meal with Buffett went for a bargain basement (comparatively, of course) $18,600, according to historical data obtained by CNBC Make It from the Glide church.
The direct fundraising for Glide church is only one small part of the philanthropic work Buffett does. As the co-founder of the Giving Pledge, the octogenarian investor, currently worth $76 billion according to Forbes, has pledged to give away more than half of his wealth.
Like this story? Like CNBC Make It on Facebook.