While Washington is focused on immigration, tax experts are focused on the opposite issue: a surge in Americans moving out.
(Read more: Buying citizenship: Which nations are affordable?)
Last year, a record 2,999 Americans gave up their citizenship or terminated their long-term U.S. residency, according to new data from the U.S. Treasury Department. That was more than three times the number in 2012, and greater than the combined totals for 2011 and 2012.
Many will look at these numbers and say they are proof that America's high taxes are chasing Americans overseas. After all, many of the expatiates who have been named are wealthy or high earners—like Facebook billionaire Eduardo Saverin or pop star Tina Turner.
(Read more: Do high state taxes chase out millionaries?)
But tax experts say it would be wrong to imply that the soaring number of expatriates are driven by wealthy Americans looking to avoid taxes.
"We do not believe that the primary reason for the increase in expatriations is for political purposes or for individuals to reduce taxes," said Andrew Mitchel, an international tax attorney who has studied the expatriate numbers closely in recent years, in his blog.
(Read more: How do you become a millionaire?)
Instead, he said there are three other reasons for the recent increases in the number of expatriations.
- Increased awareness of new tax filing laws. Many people with U.S. passports who live permanently overseas have been unaware of the rules requiring all citizens to file tax returns. The UBS tax scandal and crackdown on Americans with overseas assets has made more people aware of the requirement. And they have simply given up their citizenship rather than comply.
- Growing compliance burden. A new law, known as the Foreign Account Tax Compliance Act, was enacted in 2010 and may soon require foreign financial institutions to report to the Internal Revenue Service accounts held by U.S. taxpayers. Mitchel said some banks are simply closing the accounts of U.S. clients as a result.
- Big penalties. The standard penalty for unintentionally failing to file a tax return is $10,000. But the penalty for intentionally failing to file is far more. Mitchel gives the example of an individual who has retired overseas with savings of $1 million in a foreign bank account. If they've intentionally failed to file for four years, the penalty could be $2 million.
So yes, the IRS is clearly on the minds of Americans giving up their citizenship. But it's not because taxes have gone up. It's because the requirement to file a return—no matter where you live or what you do—has become far more stringent.
—By CNBC's Robert Frank. Follow him on Twitter @robtfrank.