Using new numbers released this week from the U.S. Census, Kolko makes the following points:
- The vacancy rate for single-family homes was 10.7 percent in 2013, up from 10.6 percent in 2012 and near its 2011 peak of 11 percent. That's far above the vacancy rate during the bubble (8.6 percent in 2005) and before (7.4 percent in 2000).
- In 2013, household formation was just 321,000, much lower than the 1.2 million baseline implied by current population growth. The number of owner-occupied single-family homes actually fell by 184,000.
- At the same time, the multi-unit vacancy rate continues to normalize, dropping for the third straight year to below its 2006 level. Despite all the new multi-unit construction, apartments are filling up.
To put it simply, there are plenty of vacant homes, no new owner households are being formed, and there's not enough demand to necessitate building more new homes. Why then do real estate agents claim there is not enough supply to meet demand, and why are home prices continuing to rise? The answer is that certain segments of the market are thriving while others are stalled and certain locations are thriving while others are stalled.
Read MoreJobless claims plunge below 300K; housing starts tumble
"There are always people who want new. Also, income growth at the high end helps boost demand for the larger new homes now being built," Kolko acknowledged. "But if new construction weren't keeping up with demand, and household formation were strong, more of the vacant homes out there would get occupied—but they're not."
On the flip side, multifamily rental construction, while down for the month, is running at quarter-century highs, and the units are filling up fast. At the same time, there are 14 million single-family homes currently occupied as rentals, and those renters appear to be staying. There were just 11 million single-family rental homes as recently as 2007.
"It's proving that households are finding renting to be much more appealing than they ever thought it might be, and they're sticking with those rental homes longer than we expected," said Buck Horne, an equities analyst at Raymond James. Horne added that Kolko "makes a fair point."
Read MoreHome builder confidence hits highest level since 2005
Horne, however, focuses on particular markets, particular builders and price points.
"If you look at the big job producing markets like California, Texas and Florida, [housing] demand there is very strong," he said. "If you're looking for first-time buyers, you'll find them in Texas."
First-time buyers nationally, however, are the weakest segment of the market, as younger millennials were hardest hit during the recession. That is why some builders, like Lennar and Pulte, are focusing on move-up models rather than cheaper, entry-level homes. Lennar reported strong third quarter earnings this week, but the Miami-based builder has been focusing on prime locations, higher-priced homes, and did not overestimate demand. DR Horton, however, announced it would have to use incentives to sell its homes. It last reported it had 10,000 unsold homes, 3,100 of those already finished.
"Other builders are sitting on more than that," Horne noted. "That's a lot to be speculating with, especially after the spring selling season."
Pulte, meanwhile, is also steering away from entry-level product and, according to Horne, is willing to give up market share as long as it means optimizing cash returns on a per-house basis. As of last quarter, Pulte had less than 1,000 spec homes in all of its combined communities.
"That's one the industry's lowest ratios of spec homes," Horne said.
Read MoreMillennials start leaving parents' homes
Home builders large and small are having to re-evaluate today's tricky market. Some, like Lennar are diversifying, getting into the multifamily market, while other smaller builders are turning to townhomes in more urban settings.
"We're constantly having our expectations pulled out from under us because we think that the market might finally be in a more permanent recovery, and now we go into a slowdown again," Stephen Paul, executive vice president of Maryland-based Mid-Atlantic Home Builders, said on CNBC's "Squawk Box." "It's hard to figure out and plan."
—By CNBC's Diana Olick.