"The American [people] rightly believe that their government has broken trust with them," Romney said. "We have all given all of our most sensitive information to the IRS. We've given them those pieces of information with the clear contract that they are going to keep it private and not politicize them.
"We are seeing a setting where government has been used to punish people that it doesn't like," he said. "This can't go on. We've got to change the structure of government to prevent this from happening in the future."
Romney further criticized the president's lack of leadership for tepid economic growth. "The president's policies are not helping," he said. "You can't keep on spending more than you take in. Governments can't overcome the power of markets.
"The reality is that we are still over 7% unemployment, and jobs haven't come back the way they would have or should have," he said. "Stock prices continue to rise, in part because of virtual zero interest rates. People are rushing their money into stocks, and it's an artificial boost."
Romney also called immigration a key reform for economic growth.
"We have got to fix our immigration and get more skilled workers into this country, fix the legal immigration system and deal with the 11 million people here illegally," he said. "This can't go on and every four years become a campaign issue which keeps Hispanic voters from voting for Republicans."
(Read More: House bipartisan immigration talks lose key member)
A bipartisan approach to legislation is what's most needed, Romney said.
"I just shake my head when I see the kind of leadership that's coming out of Washington. I wish it were possible for us to see the kind of leadership that we see in homes, churches, states and cities across the country."
—By CNBC Associate Producer Elizabeth Schulze. Follow her on Twitter