While Washington is focused on immigration, tax experts are focused on the opposite issue: a surge in Americans moving out.
(Read more: Buyingcitizenship: Which nations are affordable?)
Last year, a record 2,999 Americans gave up their citizenship or terminated their long-term U.S. residency, according to new data from the U.S. Treasury Department. That was more than three times the number in 2012, and greater than the combined totals for 2011 and 2012.
Many will look at these numbers and say they are proof that America's high taxes are chasing Americans overseas. After all, many of the expatiates who have been named are wealthy or high earners—like Facebook billionaire Eduardo Saverin or pop star Tina Turner.
(Read more: Do high state taxes chase out millionaries?)
But tax experts say it would be wrong to imply that the soaring number of expatriates are driven by wealthy Americans looking to avoid taxes.
"We do not believe that the primary reason for the increase in expatriations is for political purposes or for individuals to reduce taxes," said Andrew Mitchel, an international tax attorney who has studied the expatriate numbers closely in recent years, in his blog.
(Read more: How do you become a millionaire?)
Instead, he said there are three other reasons for the recent increases in the number of expatriations.
So yes, the IRS is clearly on the minds of Americans giving up their citizenship. But it's not because taxes have gone up. It's because the requirement to file a return—no matter where you live or what you do—has become far more stringent.
—By CNBC's Robert Frank. Follow him on Twitter @robtfrank.