The fraudsters can manipulate victim's phone's caller ID so it displays the number of a local IRS office, TIGTA said.
In some cases, the fraudsters have also told victims parts of their Social Security numbers.
In cases where victims hung up, fraudsters have called back displaying a local police phone number on caller ID, TIGTA said.
Potential victims worried about their immigration status have been threatened with deportation, TIGTA said.
(Read more: Filing taxes online? How to protect yourself)
The scam has occurred in almost every state and the fraudsters have followed a uniform script, a senior TIGTA official said on a conference call with reporters.
The technology needed to manipulate caller ID displays is easily available to the public, the official said.
Major phone companies have been warned about the scam, as well as companies that provide voice-over-the-Internet call services, the official said.
Claudia Hill, a licensed tax preparer in Cupertino, California, said that in one week last month, four of her clients complained of such scam calls. Before this year, none of her clients had previously mentioned phone scams, said Hill, who said she prepares about 1,000 tax returns a year.
(Read more: Best & worst states for taxes: How's your state?)
"The IRS is not proactive enough in getting out in front of any of this mess," she said.