"If you're working in China, you don't know whether you're working on a project for the military or not," said Carter, whose decadeslong government career also included advisory roles to Republican presidents.
"There is a duty to this country," he added. "We're in debt to the society that we live in."
Carter was responding to tech investor Peter Thiel's calls for a government investigation into Google for its "seemingly treasonous decision to work with the Chinese military and not with the U.S. military."
President Donald Trump tweeted Tuesday that his administration will "take a look" into Thiel's claims. Thiel was a supporter of Trump's 2016 presidential campaign. Google has denied working with the Chinese military.
Refusing to go as far as Thiel's "seemingly treasonous" characterization, Carter told "Squawk Box" that Google made a "mistake" in not advancing its work with the Pentagon.
Google's contract with the Defense Department, which focused on artificial intelligence, expired earlier this year, and it was not renewed. However, Google, whose search engine remains blocked in China, has continued to grow its AI center in Shanghai.
Carter's comments echoed those of former Obama White House cybersecurity chief Richard Clarke, who said Thiel was right to call out Google.
"Google refused to work for the Pentagon on artificial intelligence," Clarke told CNBC on Wednesday. "If you turn around and you work on artificial intelligence in China, and you don't really know what they're going to do with that, I think there's an issue."
On Monday, Joe Lonsdale told CNBC that his fellow Palantir co-founder Thiel was "courageous" for speaking out against Google.
"Google is not a patriotic company," said Lonsdale, also a founding partner of technology investment firm 8VC. "When Google made the choice, 'We're not going to help the U.S., but we're going to continue to work in China,' it was very clear."