Tough credit is also hitting younger buyers hardest. Today's mortgage lenders require higher down payments, and while first-time buyers used to get help from their parents, that well has dried up for some, as older Americans lost much of their savings in the recent financial crisis.
First-time buyers often turn to FHA loans, the government mortgage insurer, but premiums and fees there have gone up dramatically in the past year, and FHA's share of the market has dropped accordingly.
(Read more: Construction up, but are we building too many houses?)
On the brighter side, another report from Inside Mortgage Finance shows Fannie Mae and Freddie Mac easing the doors open a bit more to first-time buyers. The share of Fannie Mae/Freddie Mac financing for first-time homebuyers hit 19.5 percent in December. That compared with 14.1 percent a year earlier.
Credit aside, there is still the simple fact that house prices shot up like a rocket in 2013, well into the double digits nationally.
"Below the surface of last year's market, a number of unsettling trends started to emerge as a result of rapid and ultimately unsustainable appreciation, setting up a bit of a mixed bag for 2014," said Zillow's chief economist, Stan Humphries.
"Affordability issues will help put the brakes on many markets that saw huge appreciation rates, like California and the Southwest, creating volatility that could potentially cause whiplash for homebuyers and sellers."
(Read more: For the billionaire who wants it all: A fully loaded home)
—By CNBC's Diana Olick. Follow her on Twitter
Questions? Comments? facebook.com/DianaOlickCNBC