Shares of Qualcomm rose as much as 2.5 percent to $66.17, while those of Broadcom fell as much as 1.3 percent to $261.55.
"The Board has concluded that Broadcom's proposal dramatically undervalues Qualcomm and comes with significant regulatory uncertainty," Qualcomm's Presiding Director Tom Horton said in a statement.
Broadcom said it would seek to engage with Qualcomm's board and management, adding that it had received positive feedback from key customers and stockholders.
"We continue to believe our proposal represents the most attractive, value-enhancing alternative available to Qualcomm stockholders and we are encouraged by their reaction," the company said.
Both companies count Apple among their top customers. Analysts have said a deal between the two would help Qualcomm settle its legal battle with the iPhone maker as Broadcom has a closer relationship with Apple.
Broadcom made an unsolicited bid last week to buy Qualcomm in an effort to become the dominant supplier of chips used in the 1.5 billion or so smartphones expected to be sold around the world this year.
Analysts said Broadcom can now raise its bid, go for a proxy fight or launch a hostile exchange offer.
"Qualcomm's 'thanks, but no thanks' response to the unsolicited bid by Broadcom isn't surprising and we would be surprised if, at this point, Broadcom didn't move forward with a proxy fight," Loop Capital analyst Betsy Van Hees told Reuters.
If Broadcom makes a hostile bid, Qualcomm's governance rules would allow the rival to submit its own slate for the entire 11-member board by the Dec. 8 nomination deadline.
The easiest option, however, would be to talk to Qualcomm's board and agree on a higher price.
"We are well-advised and know what our options are, and we have not eliminated any of those options," Broadcom Chief Executive Hock Tan told Reuters last week.
The right price for Qualcomm could be between $80 and $85 per share, and Broadcom could go up to $90, Susquehanna analyst Christopher Rolland told Reuters.
Any deal would face scrutiny from the antitrust regulators as the combined company would own the high-end WiFi business globally, analysts said.
Regulators are already scrutinizing Qualcomm's $38-billion acquisition of automotive chipmaker NXP Semiconductors.
Broadcom has indicated it is willing to buy Qualcomm irrespective of whether it closes the NXP deal.
Qualcomm now needs to convince investors that they can create more shareholder value independently, Raymond James analyst Chris Caso said.