The average sales price for all apartments in Manhattan in the second quarter fell 5.3 percent from the prior quarter to $1.68 million and the median price fell 6.4 percent to $910,000. The number of sales closed during the second quarter remained relatively flat from the first quarter at 3,342.
Read MoreA look inside NYC's $118M penthouse
"You're seeing a wider spread now between the very high end and what we'd call the middle class in New York," said Jonathan Miller of Miller Samuel.
One reason for the growing gap is new development. Most of the new construction in Manhattan is large multimillion-dollar units rather than a variety of mid-priced to high-priced units, Miller said. Foreign buyers are also fueling the binge on hyperpriced penthouses, and while overall employment growth in New York has been weak, top wages from finance remain strong, Miller said.
Read MoreWho pays the top property taxes in the NY area?
New condos are where the real price action is, since overseas buyers can't get approved at co-ops and they like new buildings. The average sales price for new development jumped 24.3 percent in the quarter to $3.5 million.
The median price of a condo with four or more bedrooms jumped 29.2 percent to $7 million.
But all those new condos are starting to pile up faster than buyers are purchasing them. Miller said inventory in Manhattan seems to have bottomed out in the fourth quarter and is starting to rise.
Listing inventory jumped nearly 14 percent from the first quarter and is now at an absorption rate of 5.1 months, up from 4.5 months in the first quarter. That's still below historical averages, but Miller said "that tells you that a lot of the good stuff has been picked over."