It is the largest fine ever levied against a stock exchange by the SEC.
"Exchanges have an obligation to ensure that their systems, processes and contingency planning are robust and adequate to manage an IPO without disruption to the market," the SEC said in a statement. Despite "widespread anticipation that the Facebook IPO would be among the largest in history with huge numbers of investors participating, a design limitation in Nasdaq's system to match IPO buy and sell orders caused disruptions to the Facebook IPO. Nasdaq then made a series of ill-fated decisions that led to the rules violations," it said.
(Read More: Facebook Shares: One Year Later)
Nasdaq agreed to pay the fine without admitting or denying the allegations, and without further apology.
"The settlement is another important step forward and follows the commission's approval in April of our plan to accommodate investors," Nasdaq OMX CEO Robert Greifeld said in a statement. "In the last year, we have carefully reviewed these events. As market leaders, we view our experiences as opportunities to learn and improve. As part of our commitment to continually improve, we have met with many market participants, engaged leading consultants and closely examined the way we execute initial public offerings."
A few weeks after the botched offering, Greifeld appeared on CNBC and apologized to investors. "We have been embarrassed, and certainly we apologize to the industry," Greifeld said.
Facebook's IPO on May 18, 2012, was marred by technical glitches that left the market makers—who facilitate trades for brokers and are crucial to the smooth operation of stock trading—in the dark for hours as to which trades had gone through.
Nasdaq said its systems had been "tested extensively" before the IPO but that the testing did not reveal the "design flaw" that caused the glitches.
(Read More: Time to 'Defriend' Facebook?)
The SEC said several members of Nasdaq's senior leadership team convened a "Code Blue" conference call at the SEC's request and, thinking that they had fixed the problem by removing a few lines of code, chose not to delay the start of secondary market trading in Facebook shares.
But they had not grasped the root cause, the SEC said. The decision to resume trading without fully understanding the problem resulted in violations of several rules, according to the SEC, including Nasdaq's own rule governing the price/time priority for executing trade orders.
Trading was delayed until 11:30 a.m. the day of the IPO. Confirmation of the initial trades didn't post until 1:50 p.m. That left more than 30,000 Facebook orders stuck in Nasdaq's system for more than two hours when they should have been promptly executed or canceled, the SEC said.
(Read More: The 10 Biggest Internet IPOs)
The SEC also said Nasdaq broke its own rules when it assumed a short position of more than three million Facebook shares in an "unauthorized error account." The exchange then covered that short position for about $10.8 million, which also violated its rules. The regulator cited three other violations of Nasdaq's own rules during the opening of trading.
The SEC also charged Nasdaq's affiliated third-party broker-dealer, Nasdaq Execution Services, with failing to maintain sufficient net capital reserves on the day of the IPO as a result of that big short position in the unauthorized account.
"This action against Nasdaq tells the tale of how poorly designed systems and hasty decision-making not only disrupted one of the largest IPOs in history, but produced serious and pervasive violations of fundamental rules governing our markets," George S. Canellos, co-director of the SEC's Division of Enforcement, said in a statement.
Nasdaq has separately agreed to pay up to $62 million to trading firms that sustained losses during the IPO.
Greifeld said Nasdaq has taken additional steps to ensure the safety of its trading platform, including changing its procedure for "the cross," which is matching buy and sell orders, and deploying new global processes for changing technology.
It has also created two new positions (chief information officer and global head of market systems), an engineering team dedicated to monitoring and analyzing daily system performance, and a quality assurance organization focused on testing the trading systems.
"We recognize that the cornerstone of a market is investor confidence," Greifeld said.
Click here for the latest on the markets.