Revelations in IRS Scandal 'Chilling': Rep. Paul Ryan

House Budget Committee Chairman Paul Ryan told CNBC on Wednesday that investigating Obama administration scandals won't keep Republicans from pursuing sound fiscal policy.

In the Internal Revenue Service's targeting of conservative groups, what has been learned so far is "just chilling," Ryan said in a "Squawk Box" interview, claiming that the facts already known are damning.

"I see it as the arrogance of big government," he asserted, saying Congress is going to do its job on oversight and there's no need to appoint a special prosecutor who will come back "two years later after spending $20 million with some kind of conclusion."

(Read More: IRS Names Financial Manager New Chief Risk Officer)

Adam Jeffery | CNBC

Rep. Paul Ryan

Meanwhile, the Wisconsin Republican explained the GOP budget plan would result in a $110 billion surplus in its 10th year. He added that Congress is trying to "pre-empt austerity" with gradual and phased-in entitlement reform. "I'm concerned about the entitlement explosion—the bankruptcy of these programs."

There are no immediate "hard core" cuts in the Republican budget, he said, and it contains pro-growth economics.

"These things kick in later on down the road," he said. "The kind of spending cuts we're talking about right now aren't even actually cutting spending. It's just slowing the growth of spending increases."

"You've had 'the sequester,' which just deals with a small part of government—about 2 percent of the spending overall," he added.

Ryan, the GOP vice presidential nominee in the 2012 election, acknowledged that the federal budget deficit has fallen faster than expected, with the Congressional Budget Office projecting $642 billion of red ink for fiscal year 2013.

(Read More: Shrinking US Deficit Reduces Pressure for Budget Deal)

He argued, however, that it is the result of a temporary rush of revenues. "Fannie [Mae] and Freddie [Mac] cut a check we didn't see coming by about $62 billion," Ryan said. "And go figure, capital gains [taxes] went from 15 percent to 23.8 [percent] and the people, their behavior adjusted to it—meaning they realized their gains in 2012."

The U.S. needs to get on par with the rest of the world on corporate taxes, Ryan also noted, saying that Republicans want to get U.S. corporate tax rates down from 35 percent.

The House Ways and Means Committee is drafting legislation, he said, "to make sure we have an international tax system that is in sync with the rest of the industrialized world." The average global corporate tax rate is 25 percent, he added.

Ryan said he wants to make it so U.S. companies can bring "money back when you make money overseas, so that you can reinvest it in America."

Last week, Apple CEO Tim Cook made no apology for the iPhone and iPad maker saving billions of dollars in U.S. taxes through Irish subsidiaries and told lawmakers at a hearing that his company backs corporate tax reform, even though it may end up paying more.

(Read More: US Senate Panel Hammers Apple Over Offshore Tax Strategies)

"I agree with Tim Cook. I think that's a poster child for corporate tax reform," Ryan told CNBC. "Repatriation ought to be a daily event at a company's choosing."

Current tax policy also does damage to jobs and capital formation, he said, adding that the Federal Reserve's $85 billion a month bond-buying program has destroyed savers in the U.S.

—By CNBC's Matthew J. Belvedere. Follow him on Twitter @Matt_SquawkCNBC.