IRS Takes a Closer Look at the Wealthy

Rubberball | Mike Kemp | Getty Images

News that the Internal Revenue Service scrutinized conservative political groups has put the spotlight on another IRS target: the wealthy.

Audits for wealthy taxpayers have surged in recent years, leading some conservatives to question whether the IRS is unfairly focusing on those with high incomes. According to IRS data, taxpayers making $1 million or more are more than 12-times more likely than the rest of the population to be examined.

The IRS said for its 2011 examinations of 2010 returns, about one in a hundred Americans were audited. The IRS, however, audited 3.8 percent of returns for those making $200,000 or higher. The agency audited 12.5 percent of returns for those making $1 million or more.

Arthur Laffer, the conservative anti-tax crusader, said the IRS should not be picking audit targets based on their income levels.

"They should not be auditing people based on profit maximization," Laffer said. "It means that people who skim below the top levels won't be audited and it's tax cheats galore." He added that giving the IRS audit power gives it a "very powerful tool that they can use in the political process."

The IRS didn't immediately comment.

Yet former IRS officials and tax experts said audits of the wealthy are driven by economics and statistics—not politics.

(Read More: IRS Scrutiny Targeted Political Advocacy Groups)

Eric J. Toder, a co-director of the non-partisan Tax Policy Center and former director of research at the IRS, said, wealth audits and the controversy over conservative groups are "two totally different things."

The issue with conservative groups "has nothing to do with raising money, at least not directly," while audits are aimed at insuring that taxpayers pay their required amounts of tax.

"To suggest that there is a political motive behind audits of the wealthy, you'd have to see some evidence that they're targeting people with a particular political view among the wealthy. Wealth as a criteria is not a political criteria."

The IRS looks for several characteristics—not just wealth—when determining audits, he said. Filers with complex returns and business income tend to have the most errors, and since the wealthy often have business income, they attract more audits.

He added, the wealthy also generate the highest returns for auditors. According to the IRS, its examinations of people earning $1 million or more generated $4.8 billion in additional revenue, averaging about $117,000 per return.

(Read More: IRS to Conservative Groups: Sorry!)

"It doesn't serve anyone in terms of IRS or costs to put someone through an audit and then find they don't owe anything or they owe a small amount," he said.

And tax experts say audits of the wealthy will only increase in the coming years. In 2009, the IRS created a special group targeting the ultra-wealthy, called the Global High Wealth Group, staffed by experts in closely held companies, trust and estates law and overseas accounts. Tax attorneys say that after a four year ramp-up, the group has now gotten more aggressive.

The IRS has also started highlighting the group to tax attorneys and wealthy taxpayers.

"The IRS is clearly sending a message and beating the drum about this," said Charles M. Ruchelman, a tax attorney and partner at Caplin & Drysdale in Washington. Wealthy taxpayers, he said, "need to start thinking about this and being prepared."

That includes getting documents for the past three years of transactions and looking more closely at their returns.

—BY CNBC's Robert Frank. Follow him on Twitter: @robtfrank