The exact amount of damages to be awarded was unclear. Cote ordered the FHFA to submit a proposed judgment with updated damages figures based on her ruling by Friday.
But Cote, citing figures previously submitted, said the FHFA was entitled to $624.4 million, minus over $178 million in payments it received since launching the lawsuit in 2011.
The FHFA, which has acted as conservator for Fannie and Freddie since the government took them over in 2008, welcomed the ruling, which nonetheless appeared to deliver less than the $1 billion it sought at trial.
Read MoreAnother Fannie Freddie bailout? No way
"It is clear the court found that the facts presented by FHFA were convincing," FHFA General Counsel Alfred Pollard said in a statement.
Tokyo-based Nomura said in a statement that it plans to appeal, saying it was "confident that it was consistently candid, transparent and professional in all of its dealings with Fannie Mae and Freddie Mac."
A spokeswoman for RBS declined comment.
The lawsuit was the first to reach trial out of 18 the regulator filed in 2011 over some $200 billion in mortgage-backed securities that various banks sold Fannie Mae and Freddie Mac.
The FHFA previously obtained nearly $17.9 billion in settlements with banks including Bank of America, JPMorgan Chase and Deutsche Bank. Those deals followed a series of adverse rulings by Cote.
The seven deals at issue in Nomura's case were issued from 2005 to 2007. Nomura was the securities' sponsor, and RBS underwrote four of the deals.
In her ruling, Cote said the securities sold to Fannie Mae and Freddie Mac "were supported by loans for which the underwriting process had failed," with deals comprised of 45 percent to 59 percent "materially defective" mortgages.
Read MoreFannie Mae and Freddie Mac must not die
Nomura's lawyers countered that any losses incurred by Fannie and Freddie were not the banks' fault and were due to overall market conditions.
But Cote said the banks "have not quantified the loss that they say is due to macroeconomic factors."
Nor did the banks deny that "there is a link between the securitization frenzy associated with those shoddy practices and the very macroeconomic factors that they say caused the losses to the certificates," she wrote.
The case is Federal Housing Finance Agency v Nomura Holding America Inc, U.S. District Court, Southern District of New York, No. 11-06201.