The proposal, first floated by the left-leaning Fiscal Policy Institute and being crafted into legislation by State Sen. Brad Hoylman, would tax the 1,556 non-primary residences in New York City worth more than $5 million.
It would start with a 0.5 percent surcharge on the amount over $5 million and rise gradually to 4 percent for home values above $25 million. It's expected to raise about $665 million a year, according to the Institute. (Even though the tax is on New York City residents, it has to be passed by the state).
Read MoreAnd the priciest city to live in is...
Yet people who have a primary residence in New York would also have to pay the tax if they have a second home in the city valued at more than $5 million.
"If someone in New York owns more than one condo and both are valued at more than $5 million, they would pay the tax on one of the properties," said an aide to Hoylman, a Democrat.
Of course, no one's going to feel sorry for New Yorkers who own two units valued at more than $5 million, but many wealthy New Yorkers own other properties in the city as investments.
And what if an affluent family is moving within New York City? If they've purchased a new home for $5 million or more but haven't yet sold their old one, would they also face the tax? Probably.