RHB said: "There is no further development in relation to the merger at this juncture. Further updates will be made should there be any development."
Shares in CIMB rose 14.3 per cent to close at Rm5.92 while those in RHB closed up 0.90 per cent at Rm7.8.
Read MoreIt's not just oil: Trouble brews for Malaysia
In a first step towards the creation of the megabank, RHB Group was to offer CIMB shareholders 0.72 a share in RHB for every share held in CIMB. The swap assumed a value of Rm7.267 ($2.23) per CIMB share, representing a 0.4 per cent premium to CIMB's share price on July 9, when the three lenders first announced their intention to do a deal.
That was part of a complex series of share swaps that would have resulted in CIMB shareholders ending up with 70 per cent of a new merged entity with RHB shareholders owning the rest.
However CIMB's shares have fallen 18.2 per cent since the deal was announced, making it less attractive for RHB shareholders to contemplate.
More from the Financial Times:
Government borrowing costs tumble
China misses trade growth target
US energy: Off the grid
Analysts have been skeptical about the protagonists' cost savings targets, in particular those of CIMB, which has one of the highest cost-to-income ratios among Malaysian banks.
"As the share price performance indicated after the deal announcement, the market did not believe in the achievability of the synergies," said Kevin Kwek, banking analyst at Bernstein Research in Singapore.
The development is a blow to the region's aspirations to create banking groups with the financial heft to help drive the further economic integration of the 10-nation bloc known as the Association of Southeast Asian Nations (ASEAN). The countries of Asean are expected to grow by a combined average of 5.3 per cent this year, according to the Asian Development Bank.
Larger banks, including Maybank, are increasingly benefiting from the emergence of companies with ambitions to expand beyond their home markets, ahead of the creation of the so-called Asean Economic Community later this year.
The failure of the deal also scuppers the creation of what would have been a new "mega-Islamic bank". Malaysia, Southeast Asia's third-largest economy after Indonesia and Thailand, is by far the largest center in Asia for Islamic finance, driven by its majority Muslim population and strong financial and political links with the Middle East.