Aspen's outspoken former mayor, Mick Ireland, told The Associated Press that his town has become "hollowed out" because so many of the multimillion-dollar homes there are owned by super-rich owners who never use them. That's led to a town of ghost mansions.
Read MoreWhere millionaires are pouring money
"It's a mirror image of Detroit, where wealth, not poverty, is driving population down," said Ireland, who served for three terms.
For many of the super-rich owners in town, Aspen isn't their second home—it's their fourth or fifth. So for much of the year, the town is empty.
Aspen's soaring real-estate prices have led to a split population: the absentee owners who own more and more of the homes in town, and the household staff, teachers, doctors, builders and lawyers who actually work there and who often have to live more than a half-hour away to find affordable homes.
Read MoreManhattan apartment prices hit record high
Aspen does have an affordable-housing program, which Ireland helped expand and which is often seen as a model for other towns. But there are not nearly enough affordable housing units to meet demand.