"It's a big problem for them," said Matthews, who is a lawyer at Caplin & Drysdale, a tax firm based in Washington. "It decreases their competitiveness, and they may have capital flight elsewhere."
The U.S. and Russia are significant trading partners, though not all transactions would be subject to withholding. Last year, the U.S. imported $27 billion in goods from Russia, which ranked 18th among importers to the U.S., according to the Census Bureau. The U.S. exported $11 billion in goods to Russia.
The withholding would expand in 2017, if there was still no information-sharing agreement. At that point, if investors sold stocks or bonds, U.S. banks would be required to withhold a 30 percent tax on the gross proceeds from those sales.
The law would also snag big global banks with subsidiaries that don't have agreements with the IRS to share information. At first the withholding could be limited to the subsidiaries. But eventually, if any part of a large global bank refused to comply with the information-sharing requirements, the entire bank would be penalized.
"That keeps an institution from deciding that it's going to register its entity in Germany but not register the entity it has in Switzerland," said Denise Hintzke of Deloitte Tax.
Read MoreObama: More sanctions if Russia disrupts election
It would also provide a tremendous disincentive for large global banks to do business in countries where they can't share information with U.S. authorities.
More than 50 countries have reached agreements with the U.S. to share tax information about U.S. account holders. The list includes countries famous for bank secrecy, such as Switzerland and the Cayman Islands.
For Russia, the penalties could be more damaging to its economy than U.S. sanctions, said Brian L. Zimbler, managing partner of the Moscow office of Morgan Lewis, an international law firm.
"If sanctions are going to be limited to certain targeted individuals and banks, where this applies to everybody in the market, yes, I think this could potentially be worse than sanctions for the Russians," Zimbler said.
The 2010 law is known as FATCA, which stands for the Foreign Account Tax Compliance Act. It was designed to encourage—some say force—foreign banks to share information about U.S. account holders with the IRS, making it more difficult for Americans to use overseas accounts to evade U.S. taxes.