That was more than twice the 231 million pounds loss estimated by analysts, according to a poll supplied by the bank.
It also said it would miss an end-2017 deadline to sell its Williams & Glyn branch network, a condition of its 2008 state rescue.
Chief Executive Ross McEwan is in the midst of a vast, multi-year restructuring of the bank, which includes asset sales, job cuts and multi-billion dollar charges to settle litigation and pay fines for historic regulatory breaches.
The losses were partly driven by a fresh 425 million pound misconduct charge and a rise in third quarter impaired loans to 144 million pounds.
The bank also reported 469 million pounds of additional restructuring costs, largely as a result of its extended struggle to sell Williams & Glyn.
But unlike rivals Lloyds Banking Group and Barclays, RBS did not record any provisions for repaying customers mis-sold payment protection insurance after taking a 450 million pound charge in the last quarter.
The bank reported higher than expected adjusted income of 3,494 billion pounds for the three months to end-September, reflecting efforts to step up lending to maintain its status as Britain's biggest corporate lender.
RBS, which succumbed to a 45.5 billion-pound state bailout during the 2007-09 financial crisis, has not made an annual profit since 2007. The government is currently sitting on a 25 billion pound-plus loss on its investment.