Fund managers used to be able to charge high fees even if their mutual funds underperformed markets, but now they need to find new ways to win back investors who have piled out of traditional actively managed funds in a big way over the past decade, research from HSBC shows.

Over the past 10 years, investors have pulled out a third of total assets under management, or $543 billion, from actively run equity funds globally and poured $660 billion into passive ones that don't require managers to pick stocks, but instead track a benchmark index, HSBC said, citing data from EPFR, a Boston-based firm that tracks fund flows. (Read More: Investors Exit Hedge Funds After Another Bad Month.)