A surprising 13 percent said they should pay between 26 percent and 30 percent and 6 percent said they should pay between 31 percent and 40 percent.
Read MoreLearning to give away money the Buffett way
It's not like the wealthy or even super wealthy are eager to pay more taxes. The study shows that the vast majority of the affluent, millionaires and ultra-rich believe they should pay less than they already do.
The average tax rate paid by the top 1 percent of taxpayers in 2011 was 23.5 percent.
Read MoreSurprise! Some NY rich may face 164% estate tax
"Most investors, regardless of wealth level, do not believe they should be paying over 25 percent in taxes," Spectrem said, "although the tax rate for many of these households is probably over 30 percent, especially when state taxes are added to their gross incomes."
Maybe it's just that the richer you are, the more you've gotten used to paying higher rates.
—By CNBC's Robert Frank.