Hedge fund investments are finally outperforming, but the public image of their managers is taking a beating.
President Barack Obama called the group "society's lottery winners" last week, noting that the top 25 earning hedge fund managers made more in 2014 ($11.6 billion) than the roughly 158,000 kindergarten teachers in the U.S. did combined ($8.5 billion).
Presidential hopeful Hillary Clinton cited the same statistic in joining Obama's call for ending carried interest tax treatment, echoing what she said in her first campaign speech: there's "something wrong" when "hedge fund managers pay lower tax rates than nurses or the truckers."
It's not just politicians attacking investment managers.
The American Federation of Teachers' president, Randi Weingarten, cited the kindergarten comparison in speeches this month, for example, and a group called the Hedge Clippers have targeted New York-area billionaires like Paul Singer, Bill Ackman and Paul Tudor Jones. They argue that the investors are hurting the poor through some investments and political donations, protesting this year at industry events and near the managers' homes and offices.
All that makes some hedge fund managers fume. (Tweet this)
One prominent investor—who, like others, asked to remain anonymous given the divisive subject—called the rhetoric "class warfare" and noted other times in history, including before World War II, when financial speculators were unfairly blamed by politicians.
"Instead of the Jews, it's the hedge fund managers," the person said.
"The top 25 athletes, real estate investors, shipowners, actors, investment managers, movie producers, authors, all share that characteristic," another annoyed hedge fund investor wrote in an email reacting to the teacher comparison.
"Investors get to vote with their feet and if they don't like the result fire the manager by redeeming," the person added. "I do not recall ever hearing of a kindergarten teacher being fired for poor performance."
The two main industry trade groups are the Managed Funds Association and the Alternative Investment Management Association; spokesmen for both declined comment. Ken Griffin, the highest-earning hedge fund manager last year at an estimated $1.3 billion, declined to comment via a spokeswoman.
Read MoreKing Ken: Recovered Citadel chief takes the hedge fund throne
Cliff Asness, the libertarian billionaire who leads AQR Capital Management in Greenwich, Connecticut, called the Alpha ranking of the top 25 hedge fund managers sensationalist.
"If people are interested in a list that's (mostly) about 'who among the people we label "hedge fund manager" started out last year wealthiest,' then go for it," Asness wrote in an online post May 5. "But if this list is misinterpreted as being significant information beyond that, as I believe it commonly is, well that's a triumph of sensationalism and bad math over sobriety."
Asness declined to comment further.
Some agreement
Some hedge fund managers agree with Obama and others that their taxes should be raised.
"As a nation, we should be embarrassed that the top 25 hedge fund managers make more than all U.S. kindergarten teachers. What absolute insanity!" one manager told CNBC.com via email. "Yet another reason to end the absurd carried interest tax loophole."
Marc Lasry, the billionaire co-founder of Avenue Capital Group and a Clinton supporter, recently told CNBC.com he would support changing carried interest.
Read MoreSome hedge fund managers are calling for higher taxes—on themselves
Still, Clinton's criticism of investors is particularly frustrating to some.
The "class warfare" critic said it was unfair for Clinton to criticize hedge fund managers for making more than teachers given that her son-in-law, Marc Mezvinsky, helps run one, and that she has made millions of dollars in paid speech fees.
"At least we take risk, create value, allocate capital, invest in distressed companies, invest in start-ups, provide liquidity to the market," the person said. "What does Hillary do? She jawbones. You can't tell me her speeches are creating any value."
Clinton's press team did not immediately respond to requests for comment.
Nothing new for Obama
Obama, who noted high hedge fund manager pay as early as 2010, is aware of the frustration.
"I've been called Hitler for doing this, or at least this is like Hitler going into Poland. That's an actual quote from a hedge fund manager when I made that recommendation," Obama said of proposing a change to carried interest tax treatment in a speech this month at Georgetown University.
That appears to be a reference to private equity fund manager Stephen Schwarzman of Blackstone Group, who in 2010 was quoted by Newsweek as saying, "It's a war. ... It's like when Hitler invaded Poland in 1939." Schwarzman later apologized for the remarks, which were actually a reference to changing the so-called enterprise value tax treatment for the sale of a business, seen by private fund managers as more punitive move than altering carried interest.
Regardless, Obama has said it's nothing personal.
"I'm not saying that because I dislike hedge fund managers or I think they're evil. I'm saying that you're paying a lower rate than a lot of folks who are making $300,000 a year," the president said at Georgetown. "You pretty much have more than you'll ever be able to use and your family will ever be able to use. There's a fairness issue involved here."
