America's millionaire population hasn't grown significantly in 10 years, according to new government data, suggesting that not everyone at the top is benefiting from the recovery.
The latest Surveys of Consumer Finance from the Federal Reserve paints the familiar picture of widening income inequality in America. The wealthiest 3 percent of households control 54.4 percent of the nation's wealth, up from 51.8 percent in 2009.
But the gains are highly concentrated at the top of the top 3 percent. And as a whole, American millionaire households—those with a total net worth of $1 million or more—have not fared as well, either in the recession or the recovery.
Read MoreWhy rich candidates can't get elected
According to the new Federal Reserve data, there were 11.53 million millionaire households in the U.S. in 2013, down from 11.98 million in 2010 and below the 11.65 million millionaire households in 2004. (The numbers are inflation adjusted).
In other words, it's been a lost decade for America's millionaire population.
Even in percentage terms, the millionaire population is the lowest in a decade. Only 9.4 percent of American households had $1 million or more in assets in 2013, down from a peak of 10.4 percent in 2004 and even below the levels in 2001.
Read MoreWealth gap widened over past decade: Report
Compared with the rest of the country, of course, millionaires are doing fine. But the declining population of millionaires through the recession shows just how devastating the downturn was even among the affluent. It is only the truly wealthy—the top 1 percent and, more importantly, the top 0.01 percent—that have benefited most from rising stocks and asset prices.
Millionaire households
| Year | Households |
| 1989 | 5,798,525 |
| 1992 | 5,328,743 |
| 1995 | 5,231,075 |
| 1998 | 6,752,655 |
| 2001 | 10,400,735 |
| 2004 | 11,654,514 |
| 2007 | 11,887,693 |
| 2010 | 11,976,837 |
| 2013 | 11,530,695 |
Source: Source: Federal Reserve; includes households with $1 million or more in net worth.
The data also raise the question about other private surveys—from Spectrem Group, Credit Suisse, Capgeminin and others—showing a steady rise in the millionaire population.
Read More5 wealthy families who lost it all
Those surveys use different methodologies of course. And they also use different definitions of "millionaires." Spectrem defines "millionaires" as households with investible assets of $1 million or more. Capgemini defines them as individuals with $1 million or more in investible assets.
So yes, America's millionaires are growing, but only if you define them a certain way.
—By CNBC's Robert Frank