Over half of Americans aren't taking this simple step to grow their wealth

If you're stashing your hard-earned money under the mattress, or considering buying gold bars to bury in the backyard, it turns out you're not alone.

Over half of Americans, 55%, say they are not participating in the stock market, according to a new poll from MetLife of over 8,000 U.S. adults over the age of 18. The survey finds that age is definitely a factor. Gen Z (ages 18 to 24) and millennials (defined here as ages 25 to 34) are opting out in far greater numbers than older Americans.

But gender also plays a role — 44% of men report they aren't investing, compared to 59% of women. And men tend to be more likely to invest in some type of mutual or index funds and stocks.

Yet when it comes to building long-term wealth, saving alone typically isn't enough. "One of the most surprising things that people don't realize about money is saving is not enough," says Ramit Sethi, the best-selling author of "I Will Teach You to be Rich"

Over the long term, the only way to "truly grow that money" is to invest it, Sethi tells CNBC Make It.

When it comes to investing, many Americans actually are putting money into the stock market — even if they don't know it. That's because the money that you put into your retirement accounts, be it a 401(k) or an individual retirement account like a Roth IRA, are typically invested in the market. About a third of respondents to MetLife's survey said they had a retirement plan.

So why the disconnect? We tend to use the "wrong language" when it comes to investing, Erin Lowry, author of "Broke Millennial Takes on Investing," tells CNBC Make It. Collectively, Americans usually say they're "saving for retirement," so they may not recognize they're actually investing.

It also doesn't help that in many cases, you're picking a target date fund in your 401(k), rather than choosing specific funds or stocks. "They never really have to think about what it is they're investing in," Lowry says.

But there are still many Americans who actually aren't investing at all. Most of the hesitation comes down to fear, Lowry says. "There's a huge intimidation factor here," she says. And many times, the so-called experts don't really help, she adds, saying there's "almost a flippancy about how intimidating it can be to start investing."

"You're putting your hard-earned money into something that you might fundamentally not trust," Lowry says. To combat that she recommends taking the time to read about the history of the market and basic investing principles. It may sound like homework but doing some research will help you understand that markets go through cycles and how to structure your investments to hold up against down swings.

Before you start researching specific stocks or sign up for a brokerage account, Lowry says you need to figure out what you're investing for. Set some financial goals.

"You can't figure out how to invest your money until you know what you're investing for," Lowry says. "That sounds really simple but often isn't."

Setting goals is so important because whatever you're investing for determines your time horizon, which is basically when you want access to your money. For example, maybe you want to buy a house in the next three years. That's not money you want to invest in the stock market because if the market experiences a downturn, you will have less time to recover. Instead Lowry recommends a CD or a high-yield savings account earning more than 1.5% interest.

If your goals are longer term, Lowry recommends starting out by buying shares of index funds or ETFs through an online brokerage such as Charles Schwab, Fidelity or TD Ameritrade. These types of investments are typically low-cost and allow you to invest in a wide range of options, as opposed to single stock. Because of that, they are generally considered less risky.

Not sure you want to do it alone? Or maybe you want to be more hands off — then it's worth considering opening a brokerage account with a robo-advisor. These online financial platforms, which include Betterment, Schwab Intelligent Portfolios, Vanguard Personal Advisor Services and Wealthfront, will create a portfolio of investments for you and keep it on track.

Of course, Lowry points out that you don't have to invest in the stock market. It's not a requirement. But understand that the stock market has returned about 10% annually over the last century. And those returns can really help bolster your savings.

"You're going to have to save so much more money to achieve the same goals because the market is helping do some of the work," Lowry says.

Do the math, she says, and calculate how much more you'd have to save before deciding to avoid the stock market entirely.

Don't miss: Credit Karma launches a high-yield savings account that earns over 20 times more than average

Like this story? Subscribe to CNBC Make It on YouTube!