As the world's two largest payment network operators, Visa and MasterCard set the level of the fees that banks charge retailers for using their networks.
Visa's shares fell as much as 11 percent and MasterCard's by as much as 6 percent on the New York Stock Exchange.
MasterCard declined to comment. Visa did not answer calls and emails asking for comment.
The so-called Durbin amendment to Dodd-Frank, named for its sponsor—Democratic Senator Richard Durbin of Illinois— was intended to reduce burdens on retailers and hopefully trickle down to consumers in the form of lower prices.
(Read more: Retailers rethink 'back-to-school' essentials)
The amendment called on the Fed to consider certain costs to banks of providing debit cards when it set the fee cap. It was praised by retailers, but aggravated banks, which said they might have to charge consumers extra for debit cards to make up for the lost revenue.
But when the Fed announced its cap, which was higher than the regulator initially proposed, retail groups protested that it let banks charge higher fees than the law intended.
The National Retail Federation and other groups said the Fed inappropriately bowed to pressure from financial industry lobbyists and looked at bank costs beyond those the Durbin amendment directed it to consider.
The retail group on Wednesday applauded the judge's decision to throw out the Fed's rule.
"From the very beginning, retailers and restaurants knew the Federal Reserve Board of Governors had grossly misapplied the swipe fee law," Mallory Duncan, general counsel at the National Retail Federation, said in a statement after the ruling.
"They failed to heed Congress' call to set fee standards that were 'reasonable' and 'proportional' to the actual cost of a transaction," Duncan said.
(Read more: Retailers: Who's making the grade?)
The head of a banking group warned of "even more chaos ahead for consumers and small banks."
"Congress ought to save families from this uncertainty by repealing this government mandated price-fixing," Richard Hunt, president of the Consumer Bankers' Association, said in a statement.
A spokeswoman for the Fed declined immediate comment on the ruling.
In afternoon trading, Visa's shares were down 5.2 percent at $181.37. Shares of MasterCard, which earlier on Wednesday had reported a better-than-expected rise in quarterly profit, recovered their losses to trade slightly up at $605.96.
The case is in Re: NACS v. Board of Governors of the Federal Reserve System, 11-cv-02075, U.S. District Court, District of Columbia.