Despite a slight pickup in home sales over the summer, Americans are feeling less optimistic about the nation's housing recovery. The overall labor market, along with personal income growth, or lack thereof, are the key drivers of this increasing bearishness. The share of consumers who think now is a good time to buy a home fell for the second consecutive month, according to a monthly survey by mortgage giant Fannie Mae.
Just 64 percent of those surveyed think now is a good time to buy, which ties the all-time survey low.
"The August National Housing Survey results lend support to our forecast that 2015 will likely not be a breakout year for housing," said Doug Duncan, senior vice president and chief economist at Fannie Mae. "The deterioration in consumer attitudes about the current home buying environment reflects a shift away from record home purchase affordability without enough momentum in consumer personal financial sentiment to compensate for it."
While employment has certainly improved in 2014, it has not improved enough; the August jobs report from the U.S. Department of Labor fell well below expectations. Some have written the month off as an outlier, while others are concerned it is a signal of slower improvements through the back half of the year.
Read MoreDon't believe weak August jobs data: Economists
Potential home sellers are also less optimistic, as the significant home price gains through 2013 have now eased significantly. Just 38 percent of those surveyed by Fannie Mae think now is a good time to sell a home, down from July's survey. A strange disconnect is that this weaker sentiment comes as mortgage interest rates hover at the lowest levels in over a year, and fewer people think those rates will rise in the near future. Mortgage applications to purchase a home fell again last week and are now nearly 12 percent below where they were a year ago, according to the Mortgage Bankers Association.
The trouble is not affordability. In fact, after weakening for over a year, housing affordability actually flattened in the second quarter of this year, according to analysts at Deutsche Bank. The rent vs. buy comparison is finally swinging in favor of buying again.
"The drivers of the flattening have been continued rent growth, a slowdown in home price growth and some easing in rates," noted the Deutsche Bank analysts.
The biggest concern is a potential spike in mortgage rates, but so far those fears have yet to materialize.
What plagues potential home buyers more than rates is credit availability. Mortgage underwriting continues to fall on the conservative side, as lenders continue to be forced to buy back failed loans; chief executives at some of the nation's largest banks have complained publicly that the guidance for potential buy-backs is still unclear and have gone so far at to claim that the government is holding back lending.
Read MoreLower rates fail to fuel home buyers
At a recent Goldman Sachs banking conference focusing on millennial home ownership, lenders noted that the biggest problem now is not among so-called sub-prime borrowers, but among borrowers with mid-range credit scores.
"While it may be some time before we see big lending to below 680 FICO borrowers, there has been a 42 percent decline in purchase money originations from 2002 to 2013 in the 680-720 bucket which seems overly conservative," noted Goldman Sachs analysts.
This is the range that many younger, millennial borrowers fall into. They are the cohort most absent from the housing recovery. The share of first-time home buyers in today's housing market is still stuck at barely one-third, far below the historical norm of 40-45 percent.
"Most panelists acknowledged the difficult time millennials had during the recent recession. Their unemployment rate was much higher than the national average, and they are seeing more part-time work when employed as well as low levels of income growth," wrote Goldman Sachs analysts in a summary of the conference.
—By CNBC's Diana Olick.