Aspen and Detroit might seem like polar opposites as communities.
Detroit is a fallen factory town hounded by poverty. Aspen is a fast-rising mountain resort feasting on the fruits of the plutonomy and the global super rich. Houses in Detroit can't be given away, while the average home price in Aspen is worth more than $5 million.
But it turns out, the Colorado town's extreme wealth and Detroit's extreme poverty give them one thing in common: a vast community of empty homes.
Aspen's outspoken former mayor, Mick Ireland, told The Associated Press that his town has become "hollowed out" because so many of the multimillion-dollar homes there are owned by super-rich owners who never use them. That's led to a town of ghost mansions.
Read MoreWhere millionaires are pouring money
"It's a mirror image of Detroit, where wealth, not poverty, is driving population down," said Ireland, who served for three terms.
For many of the super-rich owners in town, Aspen isn't their second home—it's their fourth or fifth. So for much of the year, the town is empty.
Aspen's soaring real-estate prices have led to a split population: the absentee owners who own more and more of the homes in town, and the household staff, teachers, doctors, builders and lawyers who actually work there and who often have to live more than a half-hour away to find affordable homes.
Read MoreManhattan apartment prices hit record high
Aspen does have an affordable-housing program, which Ireland helped expand and which is often seen as a model for other towns. But there are not nearly enough affordable housing units to meet demand.
Aspen is not alone in the hollowed-out housing of wealthy communities. The influx of foreign buyers and investors buying up homes have led to more and more so-called dark buildings in New York, Miami and London.
Read MoreThe 10 swankiest penthouses
The New York Times found that a quarter of the apartments in New York City are not used as primary residences. Instead, they are used as either as pieds-à-terre or investment properties that are rented. That doesn't mean all of them are empty. But it does mean that the growing share of wealth going to the wealthy is leading to a growing share of homes being owned by people who rarely use them.
Having residents who pay property taxes but don't use many local services may be a plus. But as the people of Detroit could tell you, living in a town without people can become a hollow experience.