Read MoreThese are bad days for big boys in high finance
At this year's SALT convention, though, there were indications that some very deep-pocketed investors are intent on showing that they're not the black-hatted bad guys as portrayed to the public.
Speaker after speaker followed Scaramucci's lead in not simply talking about how much money there was to be made, but the good that money could do. While there always had been a philanthropy element to SALT, the 2014 conference shifted it into overdrive.
Here was Ken Langone, the Home Depot founder talking about raising enough to pay for the education of those entering medical school.
There was David Rubenstein, discussing the Giving Pledge, a pact among the world's richest people to give away half their wealth—only Rubenstein said he planned eventually to give away all of it, estimated by Forbes at just shy of $3 billion.
David Tepper, who made $3.5 billion last year alone, spoke on the importance of giving and how it was a value his father pushed, while White House Senior Advisor Valerie Jarrett spoke of wanting to work hand in hand with the country's wealthiest to help underprivileged Americans.
"I was the only person in the private equity world to sign the Giving Pledge," Rubenstein said, pointing out that at the time there were only 40 people who signed on and now there are about 120.
Read More'Markets are not cheap,' Rubenstein says
He calls his mode of giving "patriotic philanthropy," a term that seems to fit at least in his case after he pledged to finance the entire cost of the Washington Monument repairs (the government ended up chipping in half).
"I want to repay the country for all the good fortune I've had," the head of the Carlyle Group said. "I'm trying to encourage people to do things to give back to the country."
When bringing Tepper to the stage, Scaramucci was quick to point out not only how much money the head of Appaloosa Management made for his clients but also how he has shared in the wealth.
"This is a guy in the industry who everyone looks up to," Scaramucci said of Tepper. "He's an incredibly generous guy not only with his time and money (but also) things in philanthropy."
Read MoreTepper on the market: 'Don't be too fricking long'
Others are following suit.
The list of hedge funds either donating to charities or setting up their own foundations keeps growing.
Yet the image problems remain, and they come on a number of fronts.
It's not just that hedge funds, private equity leaders and corporate titans such as Langone have become the piñatas, it's also that questions are being raised over how well they even deliver for clients.
Read MoreWho's ruling the hedge world? You'll never guess
For 2013, the equity markets were off to the races with 30 percent or more gains, while hedge funds as an industry sputtered around 8 percent.
One of the celebrity speakers at SALT joked that it was unclear whether those gathered in the room were 1 percenters—or if that was their returns for investors. On a more serious note, the New Yorker ran an extensive analysis that coincided with the conference's run date on how hedge funds "get away with it," a reference to their below-par returns.
It is perhaps fittingly ironic, then, that the ultra-rich found a friend last week in Jarrett, whose boss, Obama, fielded that "piñata" question.
Read MorePaul Ryan stakes position in war on poverty
While there remain fences to be mended, there at least seems a realization that maybe the piñata has something good inside after all.
"We're going to do whatever we can with the ... philanthropic community," Jarrett said. "Your ability to contribute and give back is tremendous. We're looking for opportunities to partner with you."