Having a big nest egg doesn't guarantee retirement worries won't keep you up at night.
Households with at least $250,000 in investable assets are more concerned with saving and investing enough for retirement than with creating a legacy for their heirs, according to the new TIAA-CREF Affluent Investor Barometer survey, which sampled 1,242 investors in March. Even among the millionaires surveyed, only 10 percent said leaving a legacy was their top priority.
Read MoreBaby boomers' retirement fears grow
"It's not surprising that the primary concern for investors is how to generate retirement income for 30 years or more," said Sean Wilson, a wealth advisor at TIAA-CREF in New York City. "People are living longer. We tell many of our clients that they can expect to live until 95 and they need a portfolio that will last."
The TIAA-CREF report echoes the anxieties revealed by other surveys of wealthy investors.
A recent Legg Mason survey found people with $200,000 in investable assets spent 475 hours a year worrying about money and said they would need $2.5 million in retirement savings to enjoy the lifestyle they have today.
Read MoreDo you REALLY need $2.5M for retirement?
"Saving for retirement is the No. 1 priority for all my affluent clients," said Michael Chadwick, a certified financial planner at Chadwick Financial Advisors in Unionville, Connecticut, who works with many small-business owners and physicians. "My wealthiest clients are concerned with estate planning, but retirement is still the focus for most of our conversations."
Better retirement planning can alleviate many of the financial worries for the affluent and ordinary investors alike. "Clients need to figure out how much they will need in retirement, all the sources of income they will have, what they should save and the income they will need to generate to reach their goals," Wilson said.
Read MoreWhere should you put your retirement money now?
The pitfalls that diminish a nest egg are also similar across income groups. "The three biggest mistakes affluent investors make, or any investor makes, are not saving enough for retirement, starting savings too late in life and being too conservative with their investments," Chadwick said.