The legislation, which funds the government through the end of September, significantly boosts military spending. It also increases funding for border security, infrastructure and efforts to fight the opioid epidemic.
"We've seen the debt ceiling limit go up $1 trillion in the last year. We're facing trillion-dollar annual deficits starting in '19, and we can see on the horizon that the federal budget may end up spending $1 trillion on interest alone in the not-too-distant future. And what do you get for interest? Nothing," said Walker, who is seeking the Republican nomination for governor of Connecticut.
He's particularly concerned about the 70 percent of the budget that's on "autopilot" — so-called mandatory spending programs like Social Security, Medicaid and Medicare.
"We have made no meaningful progress on diffusing that ticking time bomb," Walker said. "That's what we have to get to if we want to restore fiscal sanity."
It's a different story on the Republican tax plan, which he thinks will create additional economic growth and job opportunities.
That additional growth will produce some revenues, he said.
"But let's understand this ... our fiscal gap is too great to close it with growth alone."
— CNBC's Jacob Pramuk contributed to this report.