The most popular indices, including the Mei Moses Index, are based on repeat sales of artwork that has already demonstrated marketplace demand, said Korteweg.
"Sample selection bias has a first-order impact on art indices, lowering the average annual return by 28 percent, from 8.7 percent for a standard repeat sales index to 6.3 percent for selection-corrected indices," he writes, noting the risk-adjusted return, or Sharpe Ratio, also drops nearly 60 percent. "The implications are that an investor would not find it attractive to invest in a portfolio that is representative for the broad art market, unless she derives substantial non-monetary utility from owning and enjoying art."
Read MoreGoing to cash has never looked so good
Hoffman said investors who opt for large, diversified art funds would be "very lucky" to get 10 percent to 15 percent returns. "Somewhere in the 6 percent to 8 percent range is achievable with a well-managed, diversified fund. You can potentially earn double digits, but you would need to take on higher risk."
New art fund business models, however, may offer a different advantage—tax efficiency, said Beard.
"Art funds as a sector of the investment market have had a challenged history," he said, citing the 1970s proliferation, when a group of art funds that were structured like mutual funds and led by the British Rail Pension Fund launched primarily as a way to hedge inflation. By and large, the experiment failed. Virtually no such funds exist today.
The next wave of art funds was established in the late 1990s, when modern portfolio theory redefined diversification. Pension funds and endowments with infinite time horizons started allocating a sliver of their portfolio to art funds with some success, until the credit crisis revealed that art funds are more correlated with alternative asset classes than investors were led to believe.
"Now we're in cycle 3.0," said Beard, noting art funds have evolved into private equity structures that are either closely held or private syndicates that allow a small group of investors to build a collection with tax efficiency in mind.
Still others are billing themselves as full advisory art funds, mostly started by former executives of the leading auction houses who can use their networking and institutional knowledge to source promising work, create distribution channels, bypass auction houses (and thus avoid the costly transaction fees) and partner with or advise hedge funds. "That's a fairly sizeable trend," said Beard.
"Whether this is a good way to capture diversification really gets back to your objective," said Beard. "If you really want to diversity into art in a tax-efficient way, art funds can be quite beneficial."
—by Shelly K. Schwartz, special to CNBC.com