Fed should have left easy money earlier, Paul Ryan says

The Federal Reserve needs to leave the easy money policies behind and return to "sound money," House Budget Committee Chairman Rep. Paul Ryan told CNBC on Wednesday.

"Well, I thought they should have started tapering a long time ago," he said. Quantitative easing was necessary in the beginning of the crises, he said, but not later.

"Subsequent rounds of QE I think did more harm than good," Ryan added.

(Read more: Fed taper positive sign for economy, bad for bonds)

He does not envision much change after Federal Reserve Chairman Ben Bernanke steps down and Janet Yellen takes over.

"She's clearly in the dove category," he said. "So I don't expect much difference ... from where we are right now."

(Watch more: Paul Ryan came out the winner: Pro)

Even though some critics say that signature Republican achievements, such as spending caps and sequestration, took a hit in the budget deal, Ryan stands by it. When Democrats wanted to abolish caps altogether, Republicans made them agree to 70 percent.

"It's divided government. You don't get everything you want," he said. It is important to move in the right direction, he added, "even if it's incrementally."

Ryan has not made up his mind on a 2016 presidential run, he told CNBC. For now, he added, he has goals to achieve.

"My 2014 wish list is to get to more debt reduction, get comprehensive tax reform to lower tax rates to grow the economy and get the Federal Reserve to return back to the days of sound money," he said.

—By CNBC. Follow The Kudlow Report on twitter @TheKudlowReport