While market theorists have always recognized this as a potential problem for index investing, no one has been sure exactly how it would play out or when problems might arise. As long as there are enough buyers and sellers actively setting prices and trading, index funds and stocks should pose no extra risk. It's just that no one is sure exactly how many is enough.
Indeed, not everyone thinks McCarthy is right, and others point to different risks as bigger causes for concern, including the unknown impact of the way that high-frequency traders place orders.
"So we have two new factors when it comes to a potential market situation," said John Rekenthaler, vice president of research for Chicago-based Morningstar. "There are always new factors. Most of the time, new factors don't play out according to expectations."
He pointed out that two decades ago, people worried about what the impact of 401(k)s would be in the market and whether non-professional investors would be apt to sell more quickly in a downturn. The opposite turned out to be the case.
Read MoreMost absurd ETF trade of all — paying 100 basis pts for it
Even if the risk posed by index investing is rising, the growth in index funds doesn't necessarily pose a huge system risk, pointed out Sean Collins, senior director of industry and financial analysis for the Washington, D.C.-based Investment Company Institute. "The share of assets going into index funds is rising. Does that necessarily cause markets to be dysfunctional? The answer is no," he said.
He pointed out how much more diversity there is now in index investing. Much of the money flowing into index funds has been going into markets in which there hasn't been much indexing before, including emerging markets equities and bond markets.
McCarthy said investors would be wise to look at their portfolios with the emerging risk of index funds in mind. There's not much an individual investor can do to guard against the risks posed by high-frequency trading, short of bowing out of the market entirely.