Patti McConville | Getty Image

Most of America won't shed a tear for those who own higher-priced homes, especially given that the median home price in the nation has now fallen to just $174,000, but investors and homeowners alike should take note: Higher priced homes are taking a hit and the outlook for them is worse than the overall market.

That will have ramifications for recovery.

Despite the fact that just eight percent of US loans are currently jumbo, according to Inside Mortgage Finance, and that share will rise to just 10-12 percent when the conforming loan limit is lowered October 1st, high-end housing is already being hit harder than the overall market, which isn't exactly doing so well itself.

For one, weekly mortgage applications to purchase a home have been falling steadily, down 5.7 percent last week. But jumbo loan purchase applications fell 15 percent. While sales of homes below $250,000 rose nearly 25 percent in July year over year according to the National Association of Realtors (June 2010 was the end of the home buyer tax credit, so July 2010 was artificially low, still....) sales of homes over $500,000 were basically flat.

Luxury home builder Toll Brothers

TOL loading...

Quote  |  Chart  |  News  |  Profile

[TOL  Loading...      ()   ] is not faring too well either. Revenue fell 13 percent in fiscal Q3, new orders were below expectations and cancellations rose.

"We've been extremely bearish on the entire sector since the start of the year," says Robert Wetenhall at RBC Capital Markets. "There's nothing from a data set, in terms of delinquencies or home prices or volume trends that has caused us to turn more positive."

The only nice thing he can say about Toll Brothers is that they've expanded into the multi-family market, which is where all the demand is. "People prefer renting over owning now, or multifamily as opposed to buying a house, so we're seeing a big change in consumer sentiment, so I think they're doing the right things to position the company to benefit from that switch in consumer sentiment."

Demand on the low end of the housing market is boosted by investors largely buying distressed properties; they either fix up and flip the homes or rent them out, waiting for the market to recover. Higher end homes have far fewer investors and may be more sensitive to a volatile stock market, as potential buyers are more likely to be invested there.

So why should we care about this segment of the market, if it's barely 10 percent of the overall housing market? Because when I say "high-end" I'm not talking multi-million dollar homes, I'm talking about homes over $500,000, which are move-up homes. If there's no move-up market, then there can be no real recovery because all the action is taking place in the distressed market, which then artificially pulls down the national home price numbers and scares the rest of the market away.

No move-up market also means those in higher priced homes who need to sell can't, and that could push up delinquencies on jumbo and even higher-priced conforming loans. Think of all the baby boomers who need to get out of big suburban family homes they can no longer afford. Suffice it to say, we need all segments of the housing market pushing forward in order to get the full market back to health.

Questions?  Comments?  And follow me on Twitter @Diana_Olick