Silicon Valley is supposed to be a place where a couple of guys in a garage or a dorm room can start companies that change the world. It happened with Apple and Microsoft in the 1970s, AOL in the 1980s, Amazon, Yahoo, and Google in the 1990s, and Facebook in the 2000s.
But the 2010s seem to be suffering from a startup drought. People are still starting startups, of course. But the last really big tech startup success, Facebook, is 13 years old.
Until last year, Uber seemed destined to be Silicon Valley's newest technology giant. But now Uber's CEO has resigned in disgrace and the company's future is in doubt. Other technology companies launched in the past 10 years don't seem to be in the same league. Airbnb, the most valuable American tech startup after Uber, is worth $31 billion, about 7 percent of Facebook's value. Others — like Snap, Square, and Slack — are worth much less.
So what's going on? On a recent trip to Silicon Valley, I posed that question to several technology executives and startup investors.
"When I look at like Google and Amazon in the 1990s, I kind of feel like it's Columbus and Vasco da Gama sailing out of Portugal the first time," said Jay Zaveri, an investor at the Silicon Valley firm Social Capital.
The early internet pioneers grabbed the "low-hanging fruit," Zaveri suggested, occupying lucrative niches like search, social networks, and e-commerce. By the time latecomers like Pinterest and Blue Apron came along, the pickings had gotten slimmer.
But others told me there was more to the story than that. Today's technology giants have become a lot more savvy about anticipating and preempting threats to their dominance. They've done this by aggressively expanding into new markets and by acquiring potential rivals when they're still relatively small. And, some critics say, they've gotten better at controlling and locking down key parts of the internet's infrastructure, closing off paths that early internet companies used to reach a mass market.
As a result, an industry that used to be famous for its churn is starting to look like a conventional oligopoly — dominated by a handful of big companies whose perch atop the industry looks increasingly secure.
Technology giants acquire early and often
Everyone in Silicon Valley knows the story of once-great companies like Digital Equipment Corporation, Sun Microsystems, AOL, and Yahoo that were brought down by major technology shifts. Venture capitalist Phin Barnes told me that today's technology giants have carefully studied their mistakes and are determined not to repeat them.
The management teams at today's tech giants — Facebook, Amazon, Google, and Microsoft — are "much better at understanding existential risk," Barnes told me.
For Facebook, the first big test came with the introduction of the smartphone. Facebook started out as a desktop website, and the company could have easily been caught flat-footed, like Yahoo was, by the shift to mobile devices. But Zuckerberg recognized the significance of touchscreen mobile devices and pushed his engineers to make mobile apps the top priority across the company.
Zuckerberg also went on a shopping spree, snapping up companies that seemed to be building big audiences on mobile devices. In 2012, he bought Instagram, which only had a handful of employees, for $1 billion. Two years later, he bought messaging startup WhatsApp for $19 billion.
Zuckerberg was following a model pioneered by Google. In 2006, Google paid $1.65 billion for YouTube, a site that has grown into one of the internet's most popular destinations. Most important, Google bought a little-known mobile software company called Android in 2005, laying the foundation for Google's eventual dominance of smartphone operating systems.
These acquisitions proved to be hugely significant. One ranking shows WhatsApp and YouTube as the internet's top social networks after Facebook. Instagram is next on the list if you ignore Chinese sites. If these companies had remained independent, they easily could have emerged as major competitors to Google and Facebook. Instead, they became one more piece of the Google and Facebook empires.
Amazon has pursued a similar strategy. It bought online shoe store Zappos in 2009, and the next year it bought Quidsi, the company behind a popular site for new parents called Diapers.com.
Tech companies that remain independent face tough competition
Not every technology startup accepts the giants' acquisition offers. Snapchat CEO Evan Spiegel, for example, turned down a $3 billion acquisition offer from Mark Zuckerberg in 2013, then took his company, renamed Snap, public in 2017.
Facebook has responded by building its own version of many Snapchat features. Facebook-owned Instagram introduced its own version of the Snapchat's popular stories feature last year, and within six months Instagram stories had more daily users than Snapchat itself.
Instagram has also introduced a version of Snapchat's lenses, which allow people to take whimsical rabbit-ear and dog-ear selfies. Worries about competition from Instagram has put downward pressure on Snap's stock.
Yelp CEO Jeremy Stoppelman rebuffed acquisition offers from Google and Yahoo, taking the company public in 2012. Google responded by developing its own local reviews service. And — in Stoppelman's view — Google used its dominance in the search market to give its local reviews product an unfair advantage.
"Google started turning the screws on distribution and started to bury organic search results," Stoppleman told me in a June interview. Yelp pages started to appear further down in Google search results, making it harder for Yelp to attract new users. Yelp was already popular enough to thrive in the United States, but Stoppelman argues that Google's tactics hampered Yelp's efforts to expand overseas.
And the threat of stiff competition can be a powerful inducement for independent startups to sell to the incumbents. Quidsi, the company behind Diapers.com, initially rejected Amazon's overtures. Amazon responded by slashing its own diaper prices.
"At one point," writes Businessweek's Brad Stone, "Quidsi executives took what they knew about shipping rates, factored in Procter & Gamble's wholesale prices, and calculated that Amazon was on track to lose $100 million over three months in the diaper category alone." As a venture-backed startup, Quidsi couldn't sustain those kinds of losses, so the company wound up selling to Amazon in 2010.
Modern consumer technology startups need massive warchests
Classic internet startups like Yahoo, eBay, Google, and Facebook were able to launch with modest amounts of money and reach profitability within a few years.
"Mark Zuckerberg had a huge advantage with Facebook because the pressure that normal people have of building a company was replaced by the lightness of him just playing around with ideas," said Mike Maples, an investor at the firm Floodgate.
By the time Zuckerberg founded Facebook in 2004, it didn't cost very much to run a website — even one with millions of users. So Zuckerberg was able to reach profitability quickly, and as Facebook continued to grow, the site became massively profitable, giving the company plenty of money to spend on acquisitions or new initiatives.
But recent years have been different.
As investors have realized how profitable dominant technology companies can become, they've been willing to pour more and more resources into ensuring that their startups are the ones that dominate their market. And that, ironically, has made it more difficult for anyone to reach profitability.
That's the situation in the ride-sharing market, where Uber and Lyft have waged a multi-year price war that has cost Uber billions and its smaller rival hundreds of millions of dollars. A similar dynamic has emerged in markets like food delivery, where companies have spent millions to attract customers.
Another change: Incumbent technology companies increasingly control the platforms startups use to reach users.
"Facebook grew based on saying 'give me your email addresses, and I will send out emails inviting your friends to try Facebook,'" Yelp's Stoppelman told me. "Does Facebook allow that on its own platform? Hell no. They say 'pay us $4 an install and we'll help you get one user at a time and make a lot of money in the process.'"
So while the technical costs of building an online service are cheaper than ever, it has become common for companies to spend millions of dollars on advertising to get their app or service in front of potential users. And a large share of that money flows to Google and Facebook.
The nature of innovation is changing
There's something to all of these critiques, but it's also important not to overstate them. Because for all the challenges modern startups face, there's little doubt that a startup with a truly revolutionary mass-market product would find a way to reach customers. I think that ultimately, there's a lot of merit to the low-hanging fruit hypothesis: We haven't seen any big new internet companies emerge because there's a finite number of opportunities to build big, lucrative online services.
A few months ago, the internet had a lot of fun at the expense of Juicero, a startup that sold an overpriced juice-squeezing machine. The fact that this kind of gadget for the super-wealthy got funding seems like a sign that investors are struggling to develop products with more mass-market appeal.
Juicero was an extreme example. But even recent internet startups with more mainstream products — like Snap, Square, and Pinterest — aren't likely to be as revolutionary as Apple, Amazon, and Google were in their early years.
This kind of thing has happened before. In the 1950s, 1960s, and 1970s, there was an explosion of innovation in semiconductor manufacturing. But eventually, the market settled down, with a handful of big companies — Intel, Samsung, Qualcomm — dominating the market. Innovation in "Silicon Valley" didn't stop; it just moved to things other than silicon chips.
In the 1980s, great companies like Microsoft, Adobe, and Intuit were founded to make software for PCs. Those companies still make plenty of money — just like Intel does — but there isn't a lot of room for desktop PC software startups today.
We may be reaching a similar point with apps and online services. There are only so many things you can do with a web browser or a smartphone, and maybe companies like Google, Facebook, and Snap have already locked down the most important markets.
Of course, that doesn't mean innovation in Silicon Valley is going to stop. But it might look a lot different than the innovation we've seen over the last 20 years.
Take Tesla, for example. In some ways, it's a classic Silicon Valley company. It's based in Palo Alto and employs an army of programmers to design everything from its touchscreen interface to its self-driving software.
But in other ways Tesla represents a departure from the Silicon Valley norm. While Apple manufactures iPhones in China, Tesla operates its car factory in Fremont, California. Where Uber and Airbnb have avoided owning the cars and houses, Tesla spent billions of dollars on a battery factory.
So even if incumbents like Google, Facebook, and Amazon continue to dominate the market for online services, that doesn't mean they'll remain the leaders of technology innovation more broadly. Rather, innovation may shift in dramatically different directions — toward electric cars and delivery drones, for example, rather than smartphone apps. We've gotten used to thinking of Silicon Valley, the internet, and innovation as interchangeable, but the next wave of innovation might look very different from what we're used to.
Commentary by Timothy B. Lee . Follow him/her on Twitter @binarybits.
For more insight from CNBC contributors, follow @CNBCopinion on Twitter.