The idea of staying in the family home may be a comforting thought for people heading into retirement, but for many Americans it's probably a bad idea.
The median two-person household entering retirement has a net worth of a just over $300,000, with a home—or equity in one—representing more than half of their assets, according to the latest University of Michigan Health and Retirement Study, conducted in 2010.
Despite the limited resources available to them, many Americans are loathe to move to a less-expensive home or even tap their home equity for income after paying off the mortgage.
"People have a serious behavioral resistance to touching their savings or home equity in retirement," said Steven Sass, a research economist at the Center for Retirement Research at Boston College. "Three hundred thousand dollars is not much to supplement Social Security. We think a lot more retirees need to think about using their house for income in retirement."
Many should also consider reducing their housing costs to help stretch their limited savings. A recent survey of more than 3,600 people conducted by Merrill Lynch Wealth Management and research group Age Wave suggested that Americans don't always think smaller or less expensive when it comes to retirement housing.
In fact, many think bigger. Twenty six percent of respondents who had moved since retiring cited lower home expenses as the main reason, but only 51 percent of them ended up in a smaller house. In a "downsize surprise," 30 percent actually swapped into a bigger home.
Most people do it so they can better accommodate their family," said Bill Moran, senior financial advisor and senior vice president of wealth management at Merrill Lynch.
Moran crunches the numbers on different housing scenarios—such as staying in their current home, renovating it or moving to a bigger or smaller home—for his clients. While his clients are typically wealthier than average and have more options than most, it's still an emotional issue for them.
"The housing conversation involves the intellect and the emotions," Moran said. "There's nothing wrong with the emotional side taking precedence, but my job is to show my clients the intellectual side."
Read MoreSet goals before retiring
For most people, the intellect argues for cutting your housing costs. While the family home may hold a lot of fond memories, it can be a financial millstone in retirement.
Sass, at the Center for Retirement Research, recommends that retired homeowners who are struggling financially should consider either selling their home and finding a cheaper alternative or getting a reverse mortgage that allows them to remain in the house and monetize their home equity.
He calculates that someone moving from a $250,000 home to a $150,000 home would save $3,250 annually (assuming carrying costs of 3.25 percent of the house value) and earn an additional $3,000 in income (assuming a 4 percent return on the price difference minus 10 percent in moving costs).
In addition, it's better to downsize sooner rather than later. "It's a big pain to move, and people usually do it because they're under financial pressure," he said. "All the pain is today, however, and the benefits are tomorrow."
Read MoreFor retiring boomers, it's 'go time'
The other option is a reverse mortgage. People who have spent decades paying off their mortgages may not like the idea, but a reverse mortgage can provide much-needed income. Sass recommends reverse mortgages over home equity loans, which retired borrowers may have trouble repaying down the road.
Long considered a last resort for financially strapped seniors, reverse mortgages allow homeowners to get a lump-sum payment upfront or a line of credit they can tap as needed. As long as homeowners can pay the taxes and maintenance costs of their home, they never have to leave. Borrowers repay the loan when they die or sell the house. Upfront fees are steep, but with interest rates so low, the deal is particularly good now.
"Reverse mortgages get a bad rap in the market partly because people use the money for foolish things," Sass said. "The problem is not the reverse mortgage, but how people sometimes use the money."
Perhaps the best reason to tap home equity early in retirement—particularly for individuals with limited assets—is to boost your Social Security benefits. For every year you delay filing for benefits (you can start at age 62), the government increases the annual payout by 8 percent for the rest of your life. If you wait until age 70 to file, the benefit is 76 percent higher than if you filed at age 62.
Read MorePros, cons of reverse mortgages
While it depends on the health and circumstances of each individual and his or her family, waiting to file for Social Security can dramatically improve the latter years of retirement.
"If I can borrow at a fixed cost for a higher guaranteed income in the future, I'd do it every day," said Mark Cortazzo, a certified financial planner with Macro Consulting Group. "Delaying Social Security is a really valuable asset."
He also thinks people should consider downsizing their home if they have or expect to have financial pressure at some point in retirement. "A big house is a discretionary spend in my book," Cortazzo said. "It has a cost, like vacationing, and there are trade-offs involved."
He added, "The family home is probably the place where people make most of their emotional mistakes in financial planning."
The best way to avoid those mistakes is to face the issue early and plan for it.
Moran at Merrill Lynch agrees, saying that's the most important takeaway from the Merrill housing study. "Take a deep breath and do your planning so you can head into retirement in a good frame of mind," he said. "Do it early and you can do what you want to do, not what you need to do down the road."
—By Andrew Osterland, special to CNBC.com