"You have to find the land, you've got to be able to buy it and you've got to persuade someone to let you develop it. The one you hear the most about is the last one," said Paul Emrath, vice president of survey and housing policy research at NAHB.
Emrath points to stronger no-growth movements, new environmental development standards and the plain fact that so many local politicians run on platforms that include preserving neighborhoods. The problem is more acute for so-called Class A lots, which are in the most desirable locations, usually in or near to city centers.
Read MoreUS housing starts total 1.036 million in May vs. 1.10 million expected
"Where we really see the difference in A lots is the price going up. Demand is there but at what price?" added Emrath.
Land prices have actually surpassed their peak values in many markets where builders are particularly active, especially in Texas. Finished lot values in Dallas are 40 percent higher than their peak, according to John Burns Real Estate Consulting. Other markets, like Charlotte and Raleigh in North Carolina, Denver and Phoenix are also seeing new highs.
"Land prices didn't fall nearly as much as all of us thought they would," noted John Burns in a recent interview. "Land is extremely expensive, the developer is selling it for a lot of money and the only way to make money for a builder is to build a big, huge expensive home."
That is why the median sale price of a newly built home continues to soar, up more than 8 percent in April from a year ago, according to the U.S. Census. Builders are focusing on more expensive homes, because that's where they can make the most profit. Entry-level buyers are few and far between in today's housing market overall and particularly so in new construction.
"The lower price points are just not financially feasible. Believe me, if they could sell homes for $200,000 and make money doing it, they'd be doing it all day long. There is no way they can do that," added Burns.
Nearly half of land brokers surveyed by John Burns Real Estate said lot prices had increased. The biggest annual rises were about 6 to 7 percent in Miami, Seattle, Chicago, Charlotte, Raleigh, Denver and Dallas.
For the homebuilders, those who purchased land from 2010 to 2012 are in a better position than those who waited until later.
These early buyers include Lennar, Toll Brothers, D.R. Horton and Standard Pacific.
"Builders that survived the downturn with a decent balance sheet and that were able to load up on land in late 2010-2012 are by and large sitting pretty from a land standpoint," said Rick Palacios, director of research for John Burns Real Estate.
The public builders also benefit from lower costs of construction, which allow them to pay more for land. After the announcement this week that Standard Pacific and Ryland would merge, analysts noted the benefit when it comes to land, and how the current land situation could prompt more mergers.
Read MoreWhat happens when tech workers come to town
"The land environment makes it harder for builders to operate as asset-light companies and the lack of developers hinders their ability to grow quickly," wrote analysts at UBS in a note to investors.
Land prices are expected to increase, albeit more moderately over the next few years; location, however, is always key in all facets of real estate. Markets with stronger economies, and therefore stronger housing demand, will see values climb more sharply.