Invest in You: Ready. Set. Grow.

A conflict over the No. 1 rule of investing among wealthy Americans during this crisis

Key Points
  • The Dow rally last Friday attributed to anecdotal evidence of a promising coronavirus treatment had the entire market trading like a small-cap biotech stock.
  • On the same day, Berkshire Hathaway Vice Chairman Charlie Munger told the Wall Street Journal this was not a crisis made for quick, large-scale buying.
  • Many wealthy Americans have added to stock holdings since the late March lows and stocks were rallying again on Wednesday, but even the rich remain on edge and fear another short-term, large correction ahead, revealed an E-Trade survey of investors with $1 million-plus in a brokerage account.
Spencer Platt | Getty Images

The ability to have a more opportunistic view is what separates wealthy investors from much of the investing public, but right now, it is separating them just a little less.

Mike Loewengart, chief investment officer at E-Trade Financial's capital management unit, said its quarterly survey of investors shows the millionaire set and broader investor population converging in the view that the big market rebound off the March lows is not a sure thing to last.

Fifty-seven percent of investors with at least $1 million in a brokerage account expect the market to end this quarter lower, according to a quarterly E-Trade survey which was conducted April 1 to April 8 and included a sample of over 900 self-directed active investors — the millionaire data subset provided exclusively to CNBC includes 200 investors with $1 million or more of investable assets. Nineteen percent of these affluent investors expect the market to fall by 20%.

"A technical bull market had occurred during this period of the survey, but they recognize we are not out of the woods yet," said Loewengart. "There could be significant drops, a slow grind of economic data in the coming months and we'll see how it shakes out."

Any investor that looks back at market history will not only see the longer-term rebounds that have occurred, but the magnitude of some notable drops being much larger from peak to trough than the drop that occurred in the first quarter 2020.

Loewengart noted the 84% drop, from peak to trough, after 1929, and then 57% pullback during the global financial crisis, and said these historical market data points are a reason why many investors are not jumping back in with both feet.

The stimulus already announced, and announced quickly, caused investors to come back from the brink, but there are many unknowns that remain. "We are still trying to figure out a new normal, what's going to become a new normal, even after the economy is fully reopened," he said.

"The narrative for the last few years has always been buy the dip, and that had been rewarded," Loewengart said, and 31% of millionaires expect a rise this quarter, one-third of these bulls expecting a stock market gain of 20%. But he said the data shows the extent to which "that's still coming into question," even a survey that was out in the field during a big rebound for stocks. "Many of these investors are still assessing the volatility, but they know it's part of the game and part of the risk."

The market's big guns remain worried, too

Data compiled by investor Howard Marks shows that, during the two previous bear markets, the first big comeback rallies have been followed by sharp declines until a bottom was ultimately reached. "The first and second declines were followed by substantial rallies . . . which then gave way to even bigger declines," Marks, co-founder of Oaktree Capital, wrote in a memo to clients in early April amid the rebound. The market followed a similar pattern between late 2007 and early 2009 amid the financial crisis.

On Monday, Marks told CNBC, "It took seven years to get back to the 2000 highs in 2007. It took 5½ years to get back to the 2007 highs in late 2012. So is it really appropriate that, given all the bad news in the world today, we should get back to the highs in only three months? That seems inappropriately positive."

Berkshire Hathaway Vice Chairman Charlie Munger told the Wall Street Journal last Friday, a day on which the market rallied with headlines citing some limited evidence of a coronavirus treatment showing promise, that it was not time to pounce.

"I would say basically we're like the captain of a ship when the worst typhoon that's ever happened comes. We just want to get through the typhoon, and we'd rather come out of it with a whole lot of liquidity. We're not playing 'oh goody, goody, everything's going to hell, let's plunge 100% of the reserves [into buying businesses]."

The CEOs of the market's biggest companies are not sounding immediately confident. Delta Air Lines CEO said on Wednesday, "The second quarter will be worse," in a letter to his employees. CEO Jamie Dimon said on the recent J.P. Morgan earnings call about the economic reopening that investor confidence hinges on, and which has become a source of tension between the federal, state and local governments, "A rational plan to get back is a good thing to do, and hopefully it will be tuned around later, but it won't be May. You're talking about June, July, August, something like that. "

A few simple rules have long worked out over the long-term for the affluent. It's OK to have some money in cash as part of a defensive allocation, but it is often a bad decision to rush into cash after the market has already fallen. Above all, when the market does tank, decades of equities history show it has been the wise idea for long-term investors to buy into stocks at a discount. That has been been happening as the Dow Jones Industrial Average rebounded off its late-March lows, but not with widespread conviction.

Here's what you should be doing with your 401(k) during the coronavirus
What to do with your 401(k) during the pandemic

More of the wealthy are buying than selling

"The ability to always have a long-term view, when we think of these wealthy investors, we saw it in 9/11 and the financial crisis. They have a bigger foundation to tolerate these downturns," said George Walper, president of Spectrem Group, which has studied the financial advice and affluent investor markets for a long time.

Walper said his firm's survey work didn't show wealthier investors selling all of their equities in 2008-2009, and that's the case again now. A survey it conducted earlier this month shows opportunistic buying among the affluent, but only with a slight edge over more selling.

Fifteen percent of investors have sold equities in recent weeks, while a greater percentage (21%) have purchased equities to try to take advantage of lower stock prices. But the Spectrem Group survey also showed that wealthier investors are more likely to have taken both of those approaches in April, and there has been a spike among wealthy investors who have thought of firing their advisors which eclipsed the financial crisis sentiment. 

Even wealthy millennials, that group has to hopefully realize you can't panic, but it all depends on how long you might be out of work or underemployed or taking a pay cut. That's when long-term investing is nice to talk about, but in next six months hard.
George Walper
president of Spectrem Group

The Spectrem Group survey also found that 20% of investors with $10 million or more indicated that they were thinking another financial advisor would be better fit. "Even during the financial crisis people were not saying that," Walper said, though he added that in the one to two years that followed the crisis, 12% to 15% of investors did make advisor changes.

"Over 10 years this will be smoothed out, but it is hard to be comfortable with emotionally even if it is consistent in investing," Walper said. "For younger investors it is really tough to grapple with taking a long-term view."

That is especially the case for people worried about making mortgage and rent payments. It is a great long-term opportunity for them, he said, but panic buying can be as big a mistake as panic selling. "Even wealthy millennials, that group has to hopefully realize you can't panic, but it all depends on how long you might be out of work or underemployed or taking a pay cut. That's when long-term investing is nice to talk about, but in next six months hard," Walper said.

Bear market and cash considerations

Some data points in the E-Trade survey are to be expected given the economic standstill: 59% of millionaires grade the U.S. economy at a D or F right now, and 53% think we are already in a recession. The more important E-Trade survey data finds wealthier investor bearishness increasing even more quarter over quarter than bearishness among the broader investing population, and the biggest block within the wealthy investor segment are saying it will take one to two years to recover from this bear market: 46%.Twenty-two percent said three years or longer; 31% less than a year. 

Loewengart said he zeroed in on the investors believing it will be one to two years for a recovery. "That is not that long, considering the 10-year economic expansion we came off off. That can be painful, but in the context of most investors' time horizons, not that long. If you have 30 years ahead of you, then one to two years is short lived," he said. "When you look back at the global financial crisis, it fully recovered in about 18 months for a diversified portfolio."

Josh Brown: 'Don't shy away from market volatility'
Josh Brown: 'Don't shy away from market volatility'

Loewengart said it is encouraging that wealthy investors in the firm's are not indicating greater interest in going to cash or changing allocations. Those who said they would be moving out of current positions and into cash declined from Q1 to Q2, from 25% to 19%, while those who said they would move out of cash and into new positions more than doubled, from 10% in Q1 to 24% in the current quarter. Those who said they would be making overall portfolio changes declined from 29% to 20%, while the largest group of wealthy investors indicated they would be making no changes to their portfolios this quarter (42%).  

But quarter over quarter, less millionaire investors said they were looking to buy undervalued names (down from 46% to 38%) and dividend stocks (down from 41% to 24%). Positive sentiment by S&P 500 sector for the second quarter is concentrated in health care (64% of millionaires) and consumer staples (51%). 

"Everyone is bearish in a way, and that's why consumer staples jumped up so much in popularity, same as health care," the E-Trade official said.

Daniel Kimeldorf, associate managing advisor at New York City-based Altfest Personal Wealth Management, said there was "nothing like the middle weeks of March" for the volume of calls he had with clients which were a mix between those wanted to get out of market and those who wanted to put everything in, he said, and "everything in between."

Now the majority of clients are skewed towards being opportunistic but also are still wondering whether they are "crazy to think that," Kimeldorf said.

His firm's advice hasn't changed: 100% focus on the long-term and with stocks down over 30% by late March, a risky time to invest was also a time to have some confidence that three months, or three quarters later, investors would benefit from having bought. "Three years later they will be saying what a great buying opportunity it was," Kimeldorf said.

Ben Carlson of Ritholtz Wealth Management wrote last Saturday that the recent action in stocks has shown that during the worst periods the market takes an elevator both up and down.

"The bounce we saw over the past couple of weeks resembles similar activity to the ups and downs of markets in the 1930s. I don't know how any investor could be certain about anything at the moment. Maybe instead of trying to figure out how bad losses will get investors should resign themselves to the idea that market volatility will remain elevated for some time as we try to work out how bad things will get during this crisis."

Unless something has drastically changed in an individual's life, sticking with a target asset allocations could mean buying more stocks to get back to the percentage weight stocks had before the downturn, Kimeldorf said. That is rebalancing, not changing a long-term asset allocation, or in other words, buying stocks to make up for organic losses. Altfest had many clients more defensively positioned coming into this year so in some cases cash that had already been on sidelines has been put back into the market. But these moves do not mean the market bounce since late March is an all-clear signal, though Kimeldorf said the rebound is a big reason why some investors are not as scared as they were two weeks ago, or 11 years ago.

For investors who have the ability to manage their finances right now for goals rather than short-term needs — who have not lost a job or income source which requires them to dip into investments to finance their cash flow, or already have an emergency fund with 6 to 12 months of expenses — Kimeldorf said this experience will ultimately not be so dissimilar to 2008-2009, when there was a light at the end of the tunnel.

"Consumers will return. Earnings will return. Stock values will be back to the levels that increased portfolio values over the last ten years, maybe not this quarter or this year. ... In the short term, there will be volatility. We don't want to predict week to week what the market will do. Day-to-day fluctuations in the market are not a winning recipe for long-term wealth management," he said. 

Spectrem's Walper said no one expects the ride back to be swift or sudden: "People are not expecting the market [the Dow] back to 29,000 on July 4."

What the 'Predictably Irrational' author says not to do when investing during the pandemic
What investors should be doing during the pandemic