Long frustrated about the low market valuation of Nasdaq OMX Group, the exchange operator's management is debating various ideas, ranging from further diversification to radical steps like going private or eventually splitting up the company, said several people familiar with the situation.
In the past few months, the company held talks with private equity firms, including Carlyle Group and Hellman & Friedman, about a possible buyout, two of the sources said. But talks broke down over price disagreements before they got serious, they said.
Nasdaq is no longer in talks with any private equity firms about a deal, these people said, but the company's management took the interest as a sign investors in general were beginning to see that the company was undervalued.
Nasdaq and Carlyle declined to comment. Hellman & Friedman was not immediately available for comment.
Nasdaq is prepared to take even more radical steps if valuations don't rise, such as separating its trading and technology businesses down the road, one of the sources said.
"To the extent that you are two to three years out and you've segmented the businesses, and people are saying I appreciate your sum-of-the-parts analysis, but you are still one enterprise and we will value you at the lowest common denominator of your multiples, then you do some hard thinking," the person said.
Nasdaq's management has long said the company was undervalued compared to its peers. The company is considered one of the leanest exchange operators, with strong cost controls, and has diversified its business away from cash equities into less volatile and higher-margin technology and data services businesses. Nasdaq operates 23 markets, three clearinghouses and five central securities depositories in the United States and Europe.
(Read more: Nasdaq Joins the Party, Flirts With 2000 Highs)
In a bid to make its diverse revenue sources clearer to investors, it reorganized its business in January, creating two new segments — one combining its market technology and software units, and the other linking its global data and index units. Nasdaq reported $1.7 billion in revenues last year and has a market value of about $5 billion.
Thanks in part to the changes, the company's stock has been soaring in recent months. It also benefited as trading volumes picked up, and as IntercontinentalExchange's $8.2 billion bid for Nasdaq rival NYSE Euronext sparked speculation of further consolidation in the industry.
Nasdaq's shares were trading up 1.4 percent at $31 during early afternoon trading Friday, and have gained more than 34 percent in the past three months.
Nasdaq trades at about 15 times its 2012 earnings, according to Thomson Reuters data, in line with the closest competitors to its equities business.
Rivals in market technology, though, trade at an average of 23.7 times and those in market data businesses trade at 21.9 times, according to a Nasdaq presentation this month. Nasdaq competes against such established technology services firms as SunGard and Fidessa and against data providers such as Thomson Reuters and Bloomberg.
Jillian Miller, an analyst at BMO Capital Markets, this week pegged Nasdaq's takeout value at $38 per share, while Keefe, Bruyette & Woods analyst Niamh Alexander put the takeover value in a range of $33.99 to $40.10 per share.
The problem is Nasdaq has not been able to find a buyer willing to pay that much.
One of the sources familiar with Nasdaq's thinking said a private equity firm would have to offer Nasdaq a price beyond what management thinks it, as a standalone company, could reasonably deliver to its investors in the next few years.
When Nasdaq held talks with Carlyle, its stock was trading in the low $20s, but as the stock rallied into the high $20s the talks broke down, the sources said.
Also a leveraged buyout of Nasdaq, taking advantage of its strong, consistent cash flows, could be blocked by regulators, said Richard Perott, an analyst for German bank Berenberg Bank.
In particular, European regulators are unlikely to allow Nasdaq, which provides clearing through OMX, to be highly levered due to concerns about counterparty risk.
"As exchanges are central to the orderly functioning of markets, regulators would need to be comfortable with leverage levels," Perott said.
If differences in valuations between Nasdaq and its competitors in markets technology and data persist, splitting up the company might make sense in two or three years, one of the sources said.
Another way to extract value could be to take on more debt to buy back shares. But Nasdaq has said it does not want to increase its leverage as that could lead to the loss of its investment-grade rating.
Nasdaq, like all exchanges, is also taking a closer look at potential partnerships with other exchanges in the wake of the NYSE-ICE deal, one of the sources said.
That deal, however, did not directly pressure Nasdaq to take action, the person added.
"All the NYSE-ICE deal did was change the scale profile of the company, but clearly there is no overlap in terms of the businesses they are competing in," the person said.
Nasdaq could also look at a merger with another exchange, the person said, although any deal would present challenges.
A deal with the London Stock Exchange Group, which has long been thought to be a possible partner for Nasdaq, could be difficult because the two companies are of similar size, raising the question of who would be the buyer and who would be the seller, the person said.
But one large shareholder in both Nasdaq and LSE could prompt merger talks. Borse Dubai owns 18.6 percent of Nasdaq's stock, and holds a 20.6 percent stake in LSE, according to Thomson Reuters data.
Essa Kazim, chairman of Borse Dubai, is also on Nasdaq's board.
Borse Dubai was not immediately available for comment.
In the short term, Nasdaq's most likely course of action is to continue to grow through small to mid-sized deals and organic growth, further diversifying its revenue mix.
In December, for example, the firm agreed to buy Thomson Reuters' investor relations, public relations and multimedia services units for $390 million.
Nasdaq may also want to look at making acquisitions to bolster its ability to manage counterparty risk as other exchanges have done, Perott said.
Nasdaq bought a 25 percent stake in Amsterdam-based start-up The Order Machine — a cash equity and equity derivatives trading platform — in December, with the option to take a majority stake.
Several people in the exchange industry have said one reason Nasdaq's valuation is lower than most of its peers is because it lacks a major derivatives unit, like NYSE Euronext's Liffe business. Derivatives are more profitable than cash equities, and Liffe was the main reason that ICE sought to buy NYSE Euronext.
If The Order Machine is successful in taking market share in Europe from more established players, Nasdaq will have a beachhead from which to really grow in derivatives.