×

Even in the richest 3%, there's a growing wealth gap

A luxury home with swimming pool is shown in Naples, Fla.
A. Kurtz | Getty Images
A luxury home with swimming pool is shown in Naples, Fla.

America's millionaire population hasn't grown significantly in 10 years, according to new government data, suggesting that not everyone at the top is benefiting from the recovery.

The latest Surveys of Consumer Finance from the Federal Reserve paints the familiar picture of widening income inequality in America. The wealthiest 3 percent of households control 54.4 percent of the nation's wealth, up from 51.8 percent in 2009.

But the gains are highly concentrated at the top of the top 3 percent. And as a whole, American millionaire households—those with a total net worth of $1 million or morehave not fared as well, either in the recession or the recovery.

Read MoreWhy rich candidates can't get elected

According to the new Federal Reserve data, there were 11.53 million millionaire households in the U.S. in 2013, down from 11.98 million in 2010 and below the 11.65 million millionaire households in 2004. (The numbers are inflation adjusted).

In other words, it's been a lost decade for America's millionaire population.

Even in percentage terms, the millionaire population is the lowest in a decade. Only 9.4 percent of American households had $1 million or more in assets in 2013, down from a peak of 10.4 percent in 2004 and even below the levels in 2001.

Read MoreWealth gap widened over past decade: Report

Compared with the rest of the country, of course, millionaires are doing fine. But the declining population of millionaires through the recession shows just how devastating the downturn was even among the affluent. It is only the truly wealthythe top 1 percent and, more importantly, the top 0.01 percentthat have benefited most from rising stocks and asset prices.

Millionaire households

Year Households
1989 5,798,525
1992 5,328,743
1995 5,231,075
1998 6,752,655
2001 10,400,735
2004 11,654,514
2007 11,887,693
2010 11,976,837
2013 11,530,695
Source: Federal Reserve; includes households with $1 million or more in net worth.

The data also raise the question about other private surveys—from Spectrem Group, Credit Suisse, Capgeminin and others—showing a steady rise in the millionaire population.

Read More5 wealthy families who lost it all

Those surveys use different methodologies of course. And they also use different definitions of "millionaires." Spectrem defines "millionaires" as households with investible assets of $1 million or more. Capgemini defines them as individuals with $1 million or more in investible assets.

So yes, America's millionaires are growing, but only if you define them a certain way.

—By CNBC's Robert Frank