Recent high-profile investigations include probes into tax evasion by Credit Suisse Group AG and Sudan, Cuba and Iran sanctions violations by BNP Paribas, which resulted in settlements of $2.5 billion and $9 billion, respectively. IRS Criminal Investigation has also been involved in public corruption cases such as the conviction of Jesse Jackson Jr., a former U.S. Representative and the son of civil rights leader Reverend Jesse Jackson Sr., for misusing campaign funds.
"The reason they are so important is because of the tax angle," said Ronald Machen Jr., the U.S. Attorney for the District of Columbia, adding that he wished there were more agents. "When you are looking at individuals who steal money, they are always going to want to claim less on their taxes than they have in their various bank accounts."
According to data seen by Reuters, the division expects to see the number of special agents decline to 2,130 by fiscal year 2016 due to attrition, down 13 percent from this year. That is despite hiring 48 agents this year. At its peak in 1995, the agency had 3,358 agents.
Read MoreThe IRS isbotching Obamacare tax collection
Richard Weber, the division's chief, carries around a pocket-sized, laminated card with a bar graph showing how his agency has been shrinking over the years as its budget fell. Weber in an interview said his goal is to get the number of special agents back up to at least 3,000.
"We cannot continue to lose agents to retirement and not replace them. We are at the same staffing levels that we were at in the 1970s. This is not sustainable," he said.
IRS Commissioner John Koskinen in a separate interview said the lack of resources means the unit is starting fewer investigations. "There are fewer cases that you can take," Koskinen said. He did not give specific numbers.
Overall, Koskinen estimates, the IRS, through criminal investigations as well as other activities, like audits and collection efforts, brings in $50 billion to $60 billion a year for the government, or 5 to 6 times its budget of about $11 billion. He did not say how much of that was due to the investigative unit, as opposed to audits carried out by another IRS group.
The rise of the Tea Party movement, which professes minimalist government, and an IRS scandal, has made the agency's life tougher.