Emerging entrepreneurs see the public markets as "incredibly hostile," he said.
(Read More: Software Out, Hardware Start-Ups In)
"The new running theory among new entrepreneurs is never take your company public or don't do it as long as you possibly can," Andreessen said.
Andreessen, who has invested in tech companies including, Facebook, Twitter, and Pinterest, said that another reason tech companies are hesitant to go public is because of the pain still felt from the dot-com crash.
(Read More: Facebook in Talks to Buy Israel's Waze: Report)
"A lot of people got burned during the dot-com crash, so I think we are living through the other side of that, I think we are living through sort of the tech depression, is the term I've used," he said. "But at the same time I can tell you that the vast majority of the interesting new companies that are sort of at the $50 million to $100 million revenue stage,that have historically have been the source of the exciting IPOs that put growth into the market, those companies are quite simply staying private, they aren't going public."
(Read More: Signs of Another Tech Bubble: VC)
While there were about 8,800 public companies in 1997, there were about 4,100 publicly traded companies in 2012, a decline that can be contributed to a war being waged on the public markets via regulations like Sarbanes-Oaxley and Regulation FD, which were put in place to help derisk the market. But these regulations do more harm than good, Andreessen said.
"We seem to have collectively decided as a society that we want to strip risks out of the public markets, so we had the dot-com crash, we had the financial crisis, and we decided that we don't like risk anymore," he said. "So there have been a series of regulatory reforms, a series of corporate governance movements, and the result has been a huge disincentive for companies to go public, and of course the problem with that is if new companies don't go public then you get exactly what we are seeing, which is the number of public companies falling."
Because the number of public companies is falling, growth is being stripped out of the public markets, which will ultimately pave the way for a two-tiered system, he said.
"If you kill risk, you kill growth. If you kill growth you kill returns. And as a result the market is becoming a two-tiered system and I think that is bad," he said. "Funds like ours—that primarily represent, frankly, rich people, rich people's pensions, rich people's colleges and foreign countries' sovereign wealth funds—generally, are doing fine."
However, he said that on the public side, growth is being drained out of the market, and that could hurt many Americans in the long-term.
"Of course the public market is where most ordinary Americans retirement savings is held in the form of mutual funds and pension funds and if we keep with this approach of over regulation in the public markets, we're not going to have returns for peoples' retirement accounts, which means people aren't going to be able to retire," he said. "And I think that is going to be the big problem that comes out of that and I think that's where we are heading straight towards that right now."
—By CNBC's Cadie Thompson
Follow her on Twitter