* RBS expects to start Libor settlement talks soon
* Says timing on any settlement in hands of regulators
* Q3 operating profit 1.05 bln stg vs 2 mln yr ago
* CEO won't rule out sale of U.S. business Citizens
* Shares down 1.9 pct, lag European banks
LONDON, Nov 2 (Reuters) - Royal Bank of Scotland faces fines for its part in a global interest rate setting scandal, a matter the bank said on Friday it was keen to settle as soon as possible.
The part-nationalised bank is being investigated by U.S. and UK authorities over how it set Libor and other interest rates and is expected to be one of the next to settle after British rival Barclays was fined $450 million in June.
``The group expects to enter into negotiations to settle some of these investigations in the near term and believes the probable outcome is that it will incur financial penalties,'' RBS said as it reported quarterly results on Friday.
Chief Executive Stephen Hester said it was difficult to know whether RBS faces a bigger fine than Barclays, which is the only bank to settle so far. More than a dozen banks are under investigation by authorities in the United States, Europe and Asia.
Even if a fine was smaller than Barclays' penalty it would still be a ``miserable day'' for RBS, Hester said. ``It is a deeply regrettable thing....this is the sort of thing the industry has to put behind it,'' he said.
RBS said it had dismissed a number of employees for misconduct after its own investigations into interest rate setting.
Hester said the timing of a settlement is in the hands of regulators. ``We are up for settling with all and every one as soon as they are ready,'' he told reporters on a conference call.
Libor and other past mistakes are threatening to overshadow Hester's attempts to turn the bank around, which he said would be complete in the next 15-18 months.
LEANER INVESTMENT BANK
RBS made a third-quarter operating profit of 1.05 billion pounds ($1.69 billion), up from 2 million pounds in the same period the previous year, as losses from bad debts dropped.
The bank is under political pressure to further shrink its investment bank and to consider the future of its U.S. business, Citizens.
Citizens is under scrutiny because it does not fit with RBS's narrowed focus on its home market. However analysts, who estimate Citizens could fetch over 9 billion pounds, expect Hester to delay any sale until he has got the business into better shape.
Hester said on Friday that Citizens is a core asset, but left the door open for a sale.
``I've never ruled anything out and I'm not going to start ruling it out now,'' he said. However, he added that there was a ``cogent and coherent strategy'' for the bank ``to not have all its eggs in one geographic basket''.
Shares in RBS were down 2 percent to 281 pence at 1151 GMT, lagging a 0.2 percent rise across Europe's banks.
RBS set aside another 400 million pounds to compensate customers mis-sold loan insurance, bringing its total provision to 1.7 billion pounds.
The bank said it was changing the way it pays staff particularly within its UK retail arm. ``This is a move away from the sales-based approach of the past,'' RBS said.
Bad debts were 1.18 billion in the third quarter, down 12 percent from the previous three months.
Staff costs fell 5 percent. The bank has cut 9,900 staff in the past year, or 7 percent of its workforce, most notably at its investment bank.
That business saw a 2 percent dip in revenue from the previous quarter. But operating profit jumped 18 percent to 295 million pounds due to lower costs.
Taxpayers are sitting on a loss of over 20 billion pounds after Britain pumped 45 billion pounds into the bank to keep it afloat during the 2008 financial crisis.
Compensation payouts for payment protection insurance (PPI), the investigations into interest rate rigging and possible breaches of sanctions on Iran still hang over the bank's recovery.
RBS last month exited a government insurance plan to cover its riskiest loans and sold a first tranche of shares in its insurance arm Direct Line.
But a 1.65 billion pound deal to sell 316 branches to Spain's Santander demanded by European regulators was scrapped after delays. RBS has restarted the sale.