"Individual investors are a greater detriment to themselves than high frequency traders could ever be," said Barry Glassman, a certified financial planner and president at Glassman Wealth Services in McLean, Va.
These five factors, among others, are likely to have a bigger influence on portfolio performance than high-frequency trading ever will:
Read MoreHigh-frequency trading benefits investors: Advocate
From trading costs and fund expenses to advisory charges, fees can take a substantial bite out of your portfolio. In 2012, a typical household with $120,000 in mutual fund assets paid $873 in fund fees alone, according to a Lipper analysis. Worse, many consumers are unaware of the extent of the cuts.
"There are at least 110 to 120 basis points that people don't realize they're paying," said Ron Carson, founder and chief executive of Carson Wealth Management Group. "Just because you don't see it doesn't mean it's not there."
Read MoreSticker shock—Your investment fees are more than you think
"Buy and hold" is a solid strategy, but it's not synonymous with "'til death do us part."
"I can't tell you how many times we see a portfolio and I can tell you what year that portfolio was put into effect, because I know when those funds were doing well and when they stopped doing well," said Marilyn Plum, a certified financial planner and director of portfolio management at Ballou Plum Wealth Advisors in Lafayette, Calif. Fund managers and market conditions change, and investors who don't pay attention increase their risk of lagging behind.