Net Net: Promoting innovation and managing change
Net Net: Promoting innovation and managing change

Why the VIX may not be such a great fear gauge

Brendan McDermid | Reuters

The CBOE Volatility Index, a popular gauge of whether people are expecting gyrations in the market, just saw its biggest monthly plunge in nearly three years. So are we headed for a mellow September?

Not necessarily, if you ask some of the major hedge funds, who say they're torn between betting both on a stock-market downturn and on a sharp rise up in the U.S.—and can't decide which is the more compelling case to make.

And some of them are finding that the index, commonly called "," which measures the market's expectations of volatility in the coming month by tracking the prices of certain options, or the right to buy or sell in the future, on the , is of little help in finding greater clarity, even prompting some to suggest that it has been artificially low.

The VIX hit a seven-year trough of 10.28 on July 3, and, except for a jagged rise in the first part of August, hovered around 12 for much of the summer. It lingered there on Wednesday, sliding more than 1.5 percent on another strong day in the equity markets.

The surprising truth about September

Read More10 stocks hard at work during summer vacation

Given the bargain-basement levels of the VIX and the low prices for options, Goldman Sachs market research suggested that investors bet on a move up in the stock market. "We like using options to increase equity exposure or for equity replacement," analysts wrote.

At the same time, some major investors appear to perceive warning signs.

In a June 30 letter to investors, $25 billion Elliott Management warned that a low VIX suggested a level of inertia in the markets that has been driven by six years of easy-money central bank policies that was unhealthy. The firm said many institutional money managers could be caught out when quantitative easing—the Federal Reserve's monthly bond-buying program—officially ends, most likely in October.

"While extended periods of low realized volatility are not uncommon—and we have experienced numerous such episodes in the past—we are nevertheless concerned that this complacency is occurring as a result of the most extreme monetary policy in global history," wrote Elliott, noting that the firm at that point was preparing for a possible spike in the markets after a period of such modest moves.

Read MoreInvestors basically have no fear about volatility

"It is possible (just a theory)," the letter stated, "that volatility, gold prices, and long-term rates are all simply like a coiled spring waiting to burst higher at some unpredictable confluence of events and perceptions."

Other market participants argue that the VIX, a measure of implied volatility that has traded more or less in tandem with actual volatility this year, is still an accurate barometer and that U.S. stocks have continued rallying because markets here are still more attractive than their counterparts overseas or among other asset classes.

"I don't know whether it's Europe driving that or Fed policy, but there's just been so much liquidity in the U.S. stock market, that any big move that happens just sort of gets gobbled up" by additional buyers, said Brian Stutland, the chief investment officer of Equity Armor and a longtime VIX trader. But he agrees with Elliott, and added "that the longer you keep this rubber band stretched in this low-volatility regime, the bigger than snapback will become."

—By CNBC's Kate Kelly.

Read MoreMost Americans clueless about market performance