The companies said they have instead signed an agreement to create a joint venture for their operations in Nevada, where Newmont has 19 mines adjacent to Barrick's.
Barrick launched a bid for Newmont last month, but was rejected by Newmont. The chief executives of the companies have been trading insults, bringing to fore the deep personality and cultural differences between the world's two biggest gold producers.
Barrick's offer included Newmont ditching its previously announced takeover of Canada's Goldcorp Inc, which on Monday backed the joint venture plans.
"We are finally taking down the fences to operate Nevada as a single entity in order to deliver full value to both sets of shareholders," Barrick Chief Executive Officer Mark Bristow said in a statement.
Last week, a top Barrick shareholder urged the gold miner to strike an agreement for the Nevada operations. Both companies agreed to combine their Nevada assets to cut costs, but were so far unable to hammer out a solution on how it should be done.
The joint venture, which Bristow said had been more than 20 years in the making, will be operated and majority owned by Barrick. Newmont's shares rose 1 percent premarket, while Barrick's U.S.-listed shares gained about 2.5 percent.