Wealth manager Ron Carson was recently at the Kentucky Derby with a client, an owner of Kendall-Jackson Wines. A horse that finishes in a strong position can turn out to be a pretty good investment for the wealthy. Think of a horse purchased for tens of thousands of dollars as a value stock, and the stud fees, that can reach into the millions, as dividends that will pay back the owner for years. Yet no one in the economic class that Carson was hobnobbing with at the Derby would consider a horse as a true investment, even one like Curlin, which earned $60 million throughout his career and was an investment of the Kendall-Jackson family.
"The horses are the play money," Carson, founder and CEO of Carson Wealth Management said. The problem right now, though, is that these "massively wealthy" clients, as Carson described them, are making more money from their playthings than actual investments. And having much more fun during a day at the races than watching the markets move back and forth.
"The general consensus of clients and people at the Derby is that the economy is in a twilight zone. No one really understands it," Carson said. "We are in another world. We don't know what is real."
That view matches millionaires recently surveyed by CNBC. For millionaires who can't afford to dabble in Derby horses — the majority of America's wealthy — nothing in the markets is looking very good right now, and expectations for investment performance are deflated.
The percentage of American millionaires who expect the S&P 500 will finish the year in positive territory declined by 6 percent overall since the previous CNBC Millionaire Survey, in fall 2015, while the percentage of those who believe the S&P will finish at best flat, or down for the year, increased by 5 percent. The biggest jump was in millionaires who think the S&P will be flat, which increased from 25 percent to 28 percent. Less than half of millionaires (43 percent) believe the S&P 500 will finish the year up 5 percent to 10 percent.
This view of the markets leads to another downbeat conclusion about the power of investing right now among America's wealthy: Just over half of millionaires believe the value of their assets will not change this year, and that spiked from 42 percent in the previous survey to 51 percent in the late April 2016 survey.
"People are in a wait-and-see mode and won't do a lot of investing," said Tom Wynn, director of research at Spectrem Group, which conducted the Millionaire Survey for CNBC.
In Carson's view, central banking policy is the key to uncertainty among millionaires.
"The Fed has pushed things to such an extreme with rates, and there are negative rates around the world. ... Investors look around and are not sure what to do," Carson said. "They hate sitting on cash yielding zero but don't want to take too much risk right now," he said. "It's a lot of heavy lifting to get returns. If the market takes off, they are content to not catch it. They don't want to give back a big chunk of principal if the market goes south instead."
In the fall 2015 survey, the overall markets and economic outlook of millionaires was pessimistic, and the more recent results reinforced that view among the wealthy. There was an increase, while in most cases minor, in the key pessimistic findings. The percentage of millionaires who said the economy will be stronger when surveyed in April declined to 29 percent (from 34 percent in the previous survey).
Sixty-nine percent of millionaires expect a rate of investment return no better than 2 percent to 6 percent this year.
Wynn and other wealth advisors are paying more attention to the election season and, in particular, this campaign cycle's negative rhetoric.
"I'm used to getting a lot of calls about the state of the economy, but never so many about the elections," said Karen Altfest, principal advisor and executive vice president of client relations at New York City–based wealth management firm Altfest. "People are uncomfortable with the level of rhetoric," Altfest said. "We used to not even talk about elections."
There is a long history of election seasons pulling money to the sidelines — that goes for individuals and businesses.
Altfest is taking the election more seriously as an investor issue than she ever has before. In fact, her firm already has booked a New York Times campaign reporter to come in and talk with clients the week after the presidential election is over. Altfest has never done that before. Altfest has led talks herself after elections but felt that wouldn't be enough. And while she said part of the idea was born in how interesting the events on the campaign trail have been this year, she said "bringing some sanity to clients is the goal, to calm people down and say, 'We won't know everything right away. We'll have to wait and see."
Altfest said she has tried to remind clients that a portfolio that is up 2 percent to 2.5 percent this year isn't so bad. But she said some wealthy investors already made the decision to cash out. "They aren't giving up on one risk to take another. They are just saying, 'Get me out because of the economy and election," Altfest said. "Nobody ever wants to give everything back, but now they don't even want the risk of $100,000 going to $99,000."
Carson is not a big believer that elections move millionaires, or markets. "We have a ton of discussions on the election, but I can tell you, people aren't going to invest one way or the other based on whether Hillary or Trump is in office," he said. "What I'm hearing is just more that we are in uncharted territory and people are not sure how it ends, and they are really quite content to stay neutral to slightly defensively postured."
Wynn said that recent historical data from Spectrem's Affluent Index on millionaire investing sentiment does show more pessimism in the lead-up to Election Day. In 2004 the highest months for the percentage of "Not Investing" for Americans with $500,000 or more were September and October. The same was true in 2008, but that also coincided with the global economic downturn.
Altfest has picked up on one theme underlying client concerns: "Lots of people are making up their own rules, whether it's not seeing a Supreme Court nominee or another political issue."
The idea that rules are being made up does, in a sense, match Carson's take on the world and what he is hearing from clients: "It's the craziness of irresponsible central bankers and how it all ends that will drive the next bear market," he said. "We're not smart enough to know if it comes in three months or three years from now, but you need to be hedged so you can buy assets at a reasonable price, and assets are not reasonably priced right now."
Altfest said investors have been bruised by their belief that a single stock like Apple will always do well (it's the No. 1 holding among millionaires surveyed by CNBC), and many are making the mistake of thinking the "safe" way to play this market is large-cap U.S. stocks and nothing else. (Goldman Sachs moved to neutral on the U.S. stock market on Wednesday and said credit is a better bet right now.)
The most alarming move to "safety investments" that Altfest is seeing among clients, though, is people actually putting fences around their houses (country homes in the case of her New York City clients). "Clients are telling me they need $50,000 for a fence. It's not a majority of Americans, but I've heard it more than once," Altfest said.
This surprise is one reason that when Altfest does have the New York Times reporter come in to discuss the election with clients, she will invite more clients to the talk than she ever has before.
"It may not be attractive to sit in stocks that are fully valued or overvalued, but I don't believe in going minute by minute or month by month with investments. When we get a new president, I don't think the world will implode. I don't think you need to start burying food in the backyard. I don't think it will be as scary as it seems out there right now, and that view is not the best way to take care of our futures."
If you don't believe her, build a fence or buy a horse.