For most Americans, the tax increases passed in January 2013 look like ancient history. But for the wealthy, the bill for those changes is now coming due.
The euphemistically named American Taxpayer Relief Act brought the top tax rate to 39.6 percent from 35 percent in 2012. (Whether you call this "reverting" or "raising" depends on your partisan bent). The change also phased out certain personal exemptions and it limited itemized deductions.
Taken all together, the "tax relief" for most Americans will hit the top earners when they pay their tax bills this month.
The top 1 percent of earners will pay an average tax bill of $525,231, up more than $36,000 from last year, according to the nonpartisan Tax Policy Center. It will surge to $670,000 for 2014 taxes.
The top 0.1 percent will see even bigger tax bills. That group will have an average tax burden of $2.6 million for 2013 taxes, up from $2.3 million in 2012.
Read MoreThe really rich don't really mind high taxes!?!
The incomes of the top 1 percent are also going up, of course. The average income of the top 1 percent is expected to rise from $1.67 million in 2013 to just over $2 million in 2014.
But the average tax rate that top earners actually pay is also going up. The average tax rate for the top 1 percent will go from 27.2 percent for 2012 taxes to 31.4 percent for 2013 taxes, and 33.4 percent for 2014 taxes, according to the Tax Policy Center.
The top 10 percent of taxpayers paid an estimated 52.9 percent of federal taxes in 2013 and their share is expected to rise to 55.2 percent by 2017.
Read MoreSurprise! Some NY rich may face 164% estate tax
The question many are asking this month (on both the right and the left) is how much more the wealthy can be taxed?
Some Democrats are taking a cue from "Capital in the Twenty-First Century," a new book by French economist Thomas Piketty that advocates a tax rate of 80 percent kick in on annual incomes of more than $500,000, or possibly $1 million.
Conservatives contend that the tax increases in 2013 are already stifling the investment and spending of the wealthy, and rates should be lower.
Roberton Williams, the Sol Price Fellow at the Tax Policy Center, said it's difficult to know whether, or by how much, the current tax rates are effecting the spending or investment of the wealthy.
"No one knows what the tipping point is," Williams said. "We know there is a Laffer curve, which tells us that there is a rate above which revenues will fall off and people will do less of whatever is being taxed. But we don't know where that is."
Read MoreMeet the 'hand-to-mouth' wealthy
So far, the higher tax rates have increased revenue. So the notion that higher tax rates lead to lower revenue doesn't seem to be holding up.
But Williams said it's unclear how wealthy taxpayers would be spending or investing had the tax changes not taken effect.
"We don't know the counterfactual," he said. "Especially with so many things going on with the economy, maybe the wealthy would have reported even more taxable income if they hadn't raised taxes."
—By CNBC's Robert Frank.