If you think America's wealthy don't know what it's like for the average person, think again.

In fact, 77 percent of people with at least $3 million in investable assets came from middle class or lower backgrounds while 19 percent surveyed reported growing up poor, according to a new survey. As part of an annual report, U.S. Trust surveyed 684 high net worth people to hear more about how they grew so wealthy.