The new regulations attempt to stop a range of complicated transactions companies use to lower their U.S. tax bill. One measure tries to stop "hopscotch" loans, in which companies turn their foreign earnings into U.S. loans. Another rule change tightens the application of a law that says the U.S. company's shareholders must own less than 80 percent of the new, combined company.
About 50 U.S. companies have carried out so-called inversion deals over the past decade, according to the nonpartisan Congressional Research Service. Many more had been considering it.
Richard A. Gonzalez, AbbVie's chairman and chief executive, said in a statement that the U.S. tax code should be reformed because it puts U.S. companies at a disadvantage to foreign competitors.
"The unprecedented unilateral action by the U.S. Department of Treasury may have destroyed the value in this transaction, but it does not resolve a critical issue facing American businesses today," Gonzalez said.
Read MoreUS seeks 'total cooperation' from Swiss on tax dodging: Report
With the proposed takeover of Shire dead, AbbVie's board gave shareholders something to feel good about Monday.
The board authorized a new $5 billion stock buyback program and raised its quarterly cash dividend to 49 cents from 42 cents. The dividend will be paid on Feb. 13 to shareholders of record as of Jan. 15.
AbbVie's shares rose $1.04, or about 2 percent, to $54.41 in regular trading, and were up 1.5 percent to $55.30 in pre-market trading on Tuesday.
Shares of Shire rose $5.43, or 3 percent, to $184.59. The stock was trading at $185.43 ahead of the bell.