Yet a new survey suggests different reasons. A poll by U.S. Trust and Phoenix Marketing International found the No. 1 reason the wealthy don't give is lack of faith in nonprofits. Their No. 1 response was that they feared "my gift won't be used wisely."
The second reason given for not contributing more is a "lack of knowledge or connection to a charity."
And the third reason given was "fear of increased requests from others." Donors constantly tell stories of making a gift to a group and being barraged with phone calls, emails and letters from other charities asking for more. It's easier not to give at all.
(Read more: The truth behind Peter Buffett's philanthropy rant)
Of course, there may be a difference between what the wealthy say and what they do. The same survey asked financial advisors why their wealthy clients don't give more. The top response was that they "would not have enough money to leave heirs."
The second response was "not enough money for themselves" and third was "clients don't consider themselves wealthy enough to give."
U.S. Trust said advisors don't understand their clients' giving mindset. Fully 40 percent of advisors said the wealthy would give less if the estate tax were eliminated, compared with only 6 percent of the wealthy.
But this also may be a case of the advisors saying what the wealthy won't.