Given its limited resources, the IRS said that in 2014 it will stop preparing tax returns for people who need help, such as those who are elderly or disabled. The agency also plans to answer only "basic" tax law questions, and only during the normal filing season through April 15. Instead, it will direct people to the website and other automated sources of information.
"(I)t is a sad state of affairs when the government writes tax laws as complex as ours—and then is unable to answer any questions beyond 'basic' ones from baffled citizens who are doing their best to comply," Olson said in the report.
(Read more: Marriage may not be magic bullet for poverty)
In a statement in response to the taxpayer advocate report, the agency said: "The IRS continues to make progress on a number of the critical issues outlined in the Advocate's report. It is important to note that the IRS must carefully balance limited resources to meet its dual mission of providing taxpayer service and enforcing the tax laws."
Outside experts say the lack of help is a concern both for taxpayers and for the government, which may not get all the money it should if people file their taxes incorrectly.
(Read more: The class divide: Marriage as a 'luxury good')
"It's a lose-lose situation," said Nina Crimm, a professor at St. John's University School of Law and an expert on tax issues. "It's a lose situation for … the average taxpayer, and it's a lose situation for compliance and collection of revenues."
Crimm's advice: Get started on your taxes early this year, so you aren't frantically trying to reach the IRS on April 14.
"They better start calling now," she said.
—By CNBC's Allison Linn. Follow her on Twitter @allisondlinn and Google or send her an email.