He will take over the chairmanship of the pay committee "at a date to be agreed, consistent with ensuring a smooth transition", the bank added.
Sir John, who has been a supporter of some of the large pay packages Barclays doled out to previous top management, will be retiring from the board later this year after almost 10 years, two people close to the situation said.
Read More Barclays axe swinging over investment bank jobs
"I am delighted that Crawford has agreed to join the Barclays board," said Sir David Walker, Barclays' chairman. "He brings immense experience in a range of different industries, including the financial services sector, in addition to a background in strategy and the public sector."
The appointment of Mr Gillies, who holds directorships at Standard Life and Mitie Group, comes just as Barclays is heading for a clash with investors over pay at its annual meeting next week.
The bank is expected to face a protest vote after it infuriated investors by increasing last year's bonus pool by a tenth to £2.4bn despite a drop in pre-tax profits.
Influential shareholder advisory group Glass Lewis has already recommended blocking the non-binding vote on the remuneration report for last year.
More from the Financial Times:
Barclays settles care home claim
Regulators warn as risky loans soar
Barclays in £32m executive share payout
Several large investors have told the FT that they are considering opposing the remuneration report or even voting against the remuneration policy, which would carry much more weight as it is binding. They are also angry about a boost to fixed pay for top executives in response to a new European Union-wide bonus cap.
The appointment of Mr Gillies comes after shareholder advisory group Pirc urged investors to block Sir John's re-election.
The search for a new head of the remuneration committee, who is ultimately responsible for the bank's pay policies and the size of its bonus payments, started last autumn, people close to the situation said.
Read MoreUK bankers face bonus clawback in clampdown
Sir John, the longest serving member of the Barclays board, has in the past few months been working to find a successor to Sir David, who is planning to step down as chairman next year.
Barclays' chief executive Antony Jenkins unleashed a firestorm of political and investor criticism earlier this year when he announced the bonus hike despite a one-third drop in pre-tax profits.
Mr Jenkins, a retail banker who started as chief executive in 2012 with a mandate to root out what his chairman described as pay "excess" in the sector, has argued that the increase was necessary to avoid a "death spiral" of defections at Barclays' investment bank.
Shareholders have pointed to the fact that the bonus pool for 2013 is more than 2.5 times the size of the dividend payout and that the ratio of pay to income increased from 39.6 per cent to 43.2 per cent.