5 portfolio worries bigger than high-frequency trading

HFT & the investor: Who should care?

Sorry, investors, but when it comes to your portfolio woes, high-frequency traders are not Public Enemy No. 1. Not even close.

That's not to say there aren't market factors and practices that can stack the deck against ordinary investors. But advisors say traders getting beat by powerful computers using algorithms to analyze the market and execute a high volume of trades, faster, has a minor impact, comparatively.

"Almost everything that people do nowadays has more of a threat attached to it than whether a market-making robot is taking a fraction of a penny on a trade," said Joshua Brown, chief executive of Ritholtz Wealth Management in New York.

Read MoreKatsuyama vs. O'Brien - who won the fight?

"Individual investors are a greater detriment to themselves than high frequency traders could ever be," said Barry Glassman, a certified financial planner and president at Glassman Wealth Services in McLean, Va.

These five factors, among others, are likely to have a bigger influence on portfolio performance than high-frequency trading ever will:


Read More High-frequency trading benefits investors: Advocate

From trading costs and fund expenses to advisory charges, fees can take a substantial bite out of your portfolio. In 2012, a typical household with $120,000 in mutual fund assets paid $873 in fund fees alone, according to a Lipper analysis. Worse, many consumers are unaware of the extent of the cuts.

"There are at least 110 to 120 basis points that people don't realize they're paying," said Ron Carson, founder and chief executive of Carson Wealth Management Group. "Just because you don't see it doesn't mean it's not there."

Read MoreSticker shock—Your investment fees are more than you think


"Buy and hold" is a solid strategy, but it's not synonymous with "'til death do us part."

"I can't tell you how many times we see a portfolio and I can tell you what year that portfolio was put into effect, because I know when those funds were doing well and when they stopped doing well," said Marilyn Plum, a certified financial planner and director of portfolio management at Ballou Plum Wealth Advisors in Lafayette, Calif. Fund managers and market conditions change, and investors who don't pay attention increase their risk of lagging behind.

Lack of a plan

Consumers often invest haphazardly. "That's original sin—not even knowing where to begin," said Brown.

Picking an investment based solely on media buzz, or constructing a portfolio without a larger goal than making money, can mean investors aren't aware of overlaps and imbalances that could leave them exposed to particular market risks. Nor are they aware of whether those investments are suitable for larger goals such as retirement, he said.

Read MoreRegulators warn consumers to be careful with bitcoin

Chasing performance

"People like to invest in the rear-view mirror," said Brown. But past performance isn't a guarantee of future success. Studies have found that recent top performers are just as likely to underperform as to outperform in the near term, he said.

A 2013 Vanguard study of domestic mutual fund performance over 1998-2012 found only 18 percent of 1,540 actively managed funds outperformed the market over the long term—and of those, two-thirds had at least three consecutive years of underperformance.

Bad timing

Buy low and sell high? Yeah, sure. Fund-flow reports show that investors have their timing all wrong. "In most cases, funds flow in the most right at the peak, and then money flies out after [the market] has dropped," said Glassman. Not only does that have investors selling low, but pulling out of the market when things are scary also means they miss early gains and buy in high when it recovers, he said.

Read MoreMillennial 'investors' have trust issues