The LSE said in a statement that it has fundamental concerns about key aspects of the proposal.
"Accordingly, the board unanimously rejects the conditional proposal and, given its fundamental flaws, sees no merit in further engagement," the LSE said in a statement.
LSE shares were trading up 1.6% at 7,370 pence after the statement, little changed from earlier levels.
HKEX made its £29.6 billion ($36.8 billion) offer on Wednesday and it has been coolly received by shareholders so far, though analysts expect the Asian exchange to return with an improved proposal.
The LSE has also published its letter addressed to HKEX's Chairperson Laura Cha and Chief Executive Charles Li which said it was "surprised and disappointed" that the Hong Kong Exchange had published its "unsolicited proposal" within two days of making the LSE aware of its plans.
The letter, signed by LSE Chairman Don Robert, added that the HKEX merger had no strategic merit and would be a "significant backward step."
HKEX had proposed £20.45 a share in cash for the deal, as well as 2.495 newly issued HKEX shares.
Robert added that the ongoing political upheaval in Hong Kong could make any deal unattractive to shareholders before noting that the price offered by HKEX falls short of how LSE itself values the business.
"...Even assuming your proposal were deliverable, its value falls substantially short of an appropriate valuation for a takeover of LSEG, especially when compared to the significant value we expect to create through our planned acquisition of Refinitiv," it said.
Following the letter and in a subsequent response, HKEX said it: "believes that shareholders in LSEG should have the opportunity to analyse in detail both transactions and will continue to engage with them."
Bank of Communications (BOCOM) International's Hao Hong told CNBC's "Squawk Box" on Thursday that he expected the initial bid to be rejected but that the talks would continue.
"One would expect that the price tag could go up as negotiation progresses," he told CNBC by email.
The bank strategist added that the size of the deal made it "potentially unaffordable" as HKEX shareholders may be unwilling to support any heavy borrowing sought by the Hong Kong exchange.
—CNBC's Eustance Huang contributed to this report.