Here’s why the majority of hedge funds may never be able to beat the market again

Wall Street NYSE
Getty Images

Where has all the alpha gone?

One of the biggest factors that investors have to deal with when turning their philosophy into a strategy is trying to figure out where their opportunity set lies. Competition in the markets has never been stronger, making it harder than ever to earn the elusive market outperformance that so many institutional investors actively seek.

In their book, "The Incredible Shrinking Alpha," authors and investors Larry Swedroe and Andrew Berkin lay out the case — backed up with loads of academic research — that risk-adjusted outperformance, or alpha in industry-speak, is becoming harder than ever to come by these days. But the reason has nothing to do with the old theory that a dart-throwing monkey can beat the investment professionals because they're all really emperors with no clothes. In fact, it's the opposite. Professional investors are more informed, more highly educated and more competitive than ever before. Yet they are all competing for a shrinking slice of the alpha pie.

This is what author Michael Mauboussin calls the paradox of skill. Mauboussin says, "It's not that managers have gotten dumber. It's precisely the opposite. The average manager is more skillful than in past years. The paradox of skill says that when the outcome of an activity combines skill and luck, as skill improves, luck becomes more important in shaping results." How many institutional investors bother to ask themselves if the investment managers they are investing with are lucky or truly exhibit skill?

Swedroe and Berkin go on to cite the work of David Hsieh, a finance professor at Duke University's business school, who concluded that there is roughly $30 billion of available alpha for the entire hedge fund industry. It's impossible to know if this number is correct or not, but let's assume it is for the sake of argument. In 1990, there were roughly 600 hedge funds managing close to $40 billion in assets. Today there are well over 10,000 hedge funds collectively managing close to $3 trillion. That $30 billion in alpha is much easier to share when there are fewer funds. The opportunities that were once available to these funds just aren't quite there anymore.

Author and investor William Bernstein performed a study on hedge funds for his book "Skating Where the Puck Was," where he looked at the returns on a large series of hedge fund returns and ran a regression against a three-factor model to determine the source of the performance for these funds. In the initial 1998 to 2002 period, hedge funds actually earned their keep, producing alpha of 9 percent. Unfortunately, it didn't last long as investors showered these funds with capital following the severe bear market of 2000-2002. This performance chase shrunk the alpha earned by this group in the 2003-2007 period to -0.7 percent. Then from 2008-2012 alpha went negative to -4.5 percent. The increased competition and larger capital base made it nearly impossible for these funds to keep up their outperformance.

This is why rule number one for investors is to know thyself. If these professional money managers, armed with highly educated employees and billions of dollars in assets, are having a much harder time outperforming, then what does this say about the chances individuals have in this space? That's why smaller investors most likely shouldn't be competing with the best and brightest and instead focus on areas where these huge investors do not.

The above is an excerpt from Ben Carlson's new book "Organizational Alpha." Carlson is the author of the "A Wealth of Common Sense" blog and director of institutional asset management at Ritholtz Wealth Management.

For more insight from CNBC contributors, follow @CNBCopinion on Twitter.