Knowing when to cut bait and walk away from an investment

American financier Bernard Baruch made a fortune in the stock market during the early 20th century. When asked the secret of his success, he is said to have replied, "I always sold too early."

It's a clever line—so clever that modern commentators unfamiliar with Baruch occasionally attribute it to Warren Buffett. But it reminds us that market timing has always been a tricky business. Investors tend to get emotional when they see highs and lows in the market—selling on impulse rather than strategy, and thinking they can time the market.

Phil Boorman | Getty Images

If the takeaway for you is that you should hone your market-timing skills, you should think again.

For decades, research has shown that emotionally driven trading leads to a huge gap in investor performance. Burton Malkiel's classic financial tome "A Random Walk Down Wall Street" first popularized the buy-and-hold approach to avoid market-timing mistakes back in 1973. As proved time and again, vigorous trading to catch the market at just the right moment is one of the worst things an investor can do.

Good reasons to walk away

There can be good reasons to walk away from an investment, but hoping to time a price peak isn't one of them.

The real reason to sell and switch investments is to move into something better. Before selling an asset, ask yourself what you're going to replace it with.

It can be worthwhile to sell a mutual fund, especially one intended to be a core long-term holding, if its management fee and other expenses are higher than those of similar funds with the same investment objective. Replacing it with a cheaper, more diversified and more tax-efficient mutual fund or exchange-traded fund can make a lot of sense.

Read MoreConsider risk tolerance before investing

But chasing short-term performance—another way to say selling recent losers and buying recent winners—is always a bad idea. Let's look at data from research firm Morningstar. A recent winner—a fund whose performance put it in the top quartile in 2013 among portfolios with the same investment objective—had only a 56 percent chance of doing better than average in 2014, barely better than you would expect by random chance.

As for bottom-quartile funds—the recent losers—they outperformed only 48 percent of the time in 2014; again, it is not much different than flipping a coin. So the odds are, you won't do meaningfully better or worse by chasing performance and switching investments, but you'll be on the hook for tax on any capital gain—potentially taxed as short-term capital gains—that you generate from selling the old fund.

Don't get emotional

It's important to note that a significant problem with the decision to sell—especially when it's individual securities—has nothing to do with costs and instead has a lot to do with emotions.

Loss aversion, a well-known psychological behavior, is a way to say that we feel the pain of losing more strongly than we feel the pleasure of winning. Psychologists Daniel Kahneman and Amos Tversky first introduced this concept more than 30 years ago in a study that built the foundation for what would become the field of behavioral finance.

Read MoreTales of dumb investor moves

The study, titled "Prospect Theory: An Analysis of Decision under Risk," found that loss aversion "expresses the intuition that a loss of $X is more aversive than a gain of $X is attractive ... for example, most respondents in a sample of undergraduates refused to stake $10 on the toss of a coin if they stood to win less than $30."

This indicates that the aversion to losing is so strong that people will knowingly leave money on the table in order to avoid a sense of loss.

This plays a big role in investor behavior: Investors have a (bad) habit of selling winners and not letting losers go because of loss aversion rather than for logical financial reasons. Realizing a gain feels good: We think we did something right and were rewarded for it.

Realizing a loss can feel like we made a mistake and had to pay for it. It's easy to want to avoid taking the loss and hope that a losing stock will "come back" … if only you hold on.

But there's a big problem with selling winners: taxes. A strategy of locking in gains and keeping losers is certain to be tax-inefficient, and it can easily produce worse after-tax returns. Reducing tax liability is always important, and even more so since 2013, when rates on capital gains went up and a new tax on investment returns was imposed on some high earners.

Read MoreGen Y invests like 'Depression babies'

Another reason not to be eager to sell winners is that the trend can be your friend. Strong performers can keep rising, and it works in the other direction, too: Assets that have lost value may continue to do so.

Hanging on to winners allows gains to accumulate and defers taxes on them, while selling investments that aren't living up to expectations can prevent losses from mounting. Once they do mount, another quirk of human nature comes into play—one that Baruch alluded to: the tendency for stubbornness to give way to panic, leading investors to dump their holdings at a bottom.

Strategic selling

One of the few sensible reasons to sell winners is to rebalance your portfolio. Rebalancing involves disposing of portfolio holdings in asset classes that have risen in value and using the proceeds to buy more of your asset classes that have risen less in order to restore a desired balance between stocks and bonds.

Without rebalancing, you can end up taking on much more risk as more volatile holdings (stocks) make up a greater percentage of your portfolio after a surge. Rebalancing can be done periodically—say, annually or quarterly—or whenever allocations deviate from the desired mix by some set amount. For example, five percentage points in either direction. Yes, you are selling winners, but you are doing it as part of a rational and scheduled strategy.

The need to rebalance may be more acute after a significant run of outperformance by one asset class over the other. Automated investing services perform rebalancing automatically for investors. In fact, algorithms can precisely measure the drift from the ideal allocation and the tax cost of reducing that drift and balance them down to the penny.

Tax-loss harvesting is a good reason to sell a losing asset, provided you replace it with something that offers similar risk. At its most basic level, tax-loss harvesting is selling a security that has experienced a loss—and then immediately buying a correlated asset (one that provides similar exposure, ideally in the same asset class) to replace it. The strategy allows the investor to realize a loss, which can be useful to reduce or defer a tax liability, while keeping the portfolio balanced at the desired allocation.

Baruch became a Wall Street legend by making a killing in the stock market. He controlled risk and preserved his wealth by selling appreciated assets at more or less the right time.

If you think you can match his results, good luck—but remember that few people have the time, knowledge and luck to beat the market. You're better off trying to achieve the same results by selling in a systematic, disciplined way to stay properly diversified.

—By Jon Stein, founder and CEO of Betterment.