The deal "cannot proceed under its current structure," the source said.
The companies said they were still in discussions and the proposed transaction had not yet been terminated.
Shares of 3Com fell 73 cents, or 19.6 percent, in afternoon trading on Nasdaq to $3 as the possibility of the takeover proceeding seemed remote.
"We are very disappointed that we were unable to reach a mitigation agreement with CFIUS for this transaction," said 3Com President Edgar Masri.
"While we work closely with Bain Capital Partners and Huawei to construct alternatives that would address CFIUS' concerns, we will continue to execute our strategy to build a global networking leader," Masri said.
"It's better to withdraw the application than to have it rejected, but it's a bad sign that they couldn't reach an agreement in the review window," said one arbitrageur who declined to be named.
Bain agreed in September to buy 3Com in a deal that would also give Huawei Technologies a 16.5 percent minority stake. Huawei could increase its stake in 3Com by up to an additional 5 percent.
The deal was subject to scrutiny by CFIUS, an inter-agency U.S. governmental panel that reviews corporate acquisitions involving foreign buyers.
In October, eight U.S. lawmakers were backing a bill suggesting that the planned buyout 3Com "threatens the national security of the United States."
3Com previously said Huawei would not have access to sensitive U.S. technology or U.S. government sales and that it would lack operational control or the ability to make decisions for the firm.
Last week, Bain offered various concessions. including one proposal under which it would divest 3Com's Tipping Point unit, which makes national security software, a source familiar with the situation previously said.
Last year, 3Com had planned to spinoff Tipping Point through an initial public offering, but that idea was scrapped when Bain agreed to buy Marlborough, Massachusetts-based 3Com.
Tipping Point makes "intrusion prevention" systems to protect networks at large businesses and government agencies. 3Com acquired Tipping Point in late 2004 for $430 million.
A Treasury Department spokesman confirmed that Bain and 3Com withdrew their CFIUS application, but declined to comment on the specifics of the case.
In late January, U.S. President Bush issued an executive order clarifying procedures for such security reviews in the wake of several deals by sovereign wealth funds to invest billions of dollars into major U.S. banks.
Bush's order spelled out how CFIUS conducts its reviews and informs Congress, including stating that it may impose conditions on a deal to address any potential national security risks after a threat-analysis by the U.S. director of national intelligence. But the order also states that the U.S. remains open for business and welcomes foreign investment.
"The (Bush) administration has worked hard to implement the CFIUS process in a manner that ensures the protection of national security while also encouraging investment from abroad," U.S. Treasury spokesman Robert Saliterman said.