The chart is broken into three periods separated by the vertical black bars:
1. August 4 and September 18, 2014, where a steadily rising market flattens out. During this time, the S&P500 rose by roughly 5% and the positive and negative sentiment scores both oscillated around 1.
2.A sharp drop of roughly 6 percent in the S&P 500 between September 18 and October 15 2014 in response to fears of a global slowdown. The positive sentiment dropped dramatically by 60 percent from peak to trough and an average drop of over 30 percent to 0.66, while the level of negative sentiment drifted slightly higher with three pronounced peaks of increasing magnitude at weekly intervals: on September 27, October 4 and October 11 (sentiments tend to peak on Saturdays which is not altogether surprising).
3.A sharp reversal of over 8 percent for the S&P 500 was seen between October 15 and October 31, as the market made new highs. Interestingly, the average negative sentiment has stayed above 1 during the rebound while the positive sentiment has remained at 0.6, significantly lower than the pre –"global slowdown" level.
It is tempting to conjecture that the market is currently in a heightened state of anxiety. What might explain the major drop in positive sentiment?
One interpretation could be that while there are always doomsayers: when the countervailing optimists drop out of the discussion, it bodes ill for the market. A contrary interpretation, however, could be that the low positive sentiment scenario coupled with a healthy level of negative sentiment is indicative of a key segment of the investment community having already taken money off the table.
In this latter narrative, this "sidelined" money can be put back into the market to provide and support a market rise in the near term. (This could, for example, explain the recent market rise despite the low positive sentiment.) A slightly different variant of this view could hypothesize that low negative sentiment coupled with high positive sentiment portends a heightened state of risk in which investors are fully invested. However, bad news can spark a rapid correction.
The possibility of multiple theories suggests that we must dig deeper and more methodically into the data to try to address such questions and identify characteristic viruses, and to measure which ones are getting stronger or weaker.
As of this writing, for example, concerns about Europe have resurfaced to include the narratives of stress test failures of certain European banks and/or the need for aggressive quantitative easing. Could this environment of depressed positive sentiment be a fertile one, in which a pessimistic virus can take hold (particularly if a contingent of the investment community were to become nervous and take risk off the table)?
Read MoreECB survey downgrades inflation, growth forecasts
Regardless of how the future pans out, professional investors cannot ignore the emergence of a new and critical class of "factor" in their investment models—namely, big data based sentiment, and the growing capability for market participants to extract and measure it reliably.
This new type of factor, which is very different from indices such as the Michigan consumer sentiment index, may well join the ranks of key determinants of investment performance, alongside traditional factors such as value and momentum that have been used for decades on Wall Street.
Commentary by Vasant Dhar, professor at NYU Stern School of Business and editor-in-chief of Big Data Journal.