Douglas Box wishes he'd done some things differently. His three brothers probably do, too.
In 1993 the four Box brothers inherited a successful oil business when their entrepreneurial father—former Detroit Lions football player Cloyce Box—died at the age of 70. The elder Box had left no detailed plan of succession, and sibling rivalry broke out when the board of directors of the public company, Box Energy, chose a younger brother over the eldest as chief executive.
Over the next four years, the four brothers filed lawsuits against each other as the business deteriorated. They eventually were forced to sell it for far less than they might have when they had inherited the company.
"We ended up selling an oil business in 1997 … and look what happened [to the price of oil]," said Douglas Box. He is currently writing a book about his family experiences and now works as a financial advisor at the firm he founded, Box Family Advisors, to help other families avoid what he and his brothers went through.
"We got to a place where we had to sell the business," he said. "A federal judge threatened to put the company in receivership, and that was when we settled. It was crazy we did that. "
The greatest strength of a family business—namely, the human element—can also be its greatest weakness. When family members are working well together, they can help drive some of the most successful businesses in the world, Box explained, but "when families get into conflict, things can fall apart quickly."
Doug Baumoel is an advisor of last resort for families in intractable conflict. The founder of Continuity Family Business Consulting, Baumoel said about 90 percent of his family clients are either stuck from a decision-making perspective or are actively fighting with each other. Many are already in litigation.
Read MoreVet future advisors with 5 questions
"The fighting isn't usually about money; it's about identity," said Baumoel, who holds two certificates in family business and wealth advising from the Family Firm Institute. "When people's self-image is threatened, they fight back." He added, "They're in pain when they call us. "
The advisors in Baumoel's firm all boast master's degrees in business administration, and the services they provide are oriented more toward business consulting rather than mediation or financial advice.
Baumoel said that neither mediation nor communication are what his clients need most. "People are usually fighting about things that aren't negotiable, and families in conflict are often communicating just fine," he said.
Read MoreLearning secures family wealth
He said that conflict is inherent in any family business and that his objective is not to eliminate it but to help reduce it by professionalizing management of the business. "If we can improve the job descriptions, the compensation system and the business processes, we can help them grow themselves out of conflict and articulate a better structure of governance for the business," he said.
"There are many good reasons to sell a business, " Baumoel explained. "The tragedy is when a business has to be sold because of insufficient planning or infighting, " he said.
Sometimes, of course, that approach just won't work. When members of a family are unable to settle conflicts and agree on a system for governing a family business and distributing its benefits, the best option is often to sell—and the sooner, the better.
The windows for selling businesses can close quickly, and the current strong equity markets and low cost of capital present an ideal selling environment.
Read MoreDonate the tax-smart way
What's more, when family-member owners of a company are in bitter conflict, it usually results in a deterioration of the business and ultimately of the family's wealth.
"If a family has either internal or external threats that will jeopardize their business in the future, it's time to at least bring in a third party to explore the potential sale of the business," suggested Grant Rawdin, a certified financial planner and CEO of Wescott Financial Advisory Group.
In such cases, the best case scenario is one in which family members at least agree to try to extract as much value as they can from a sale of the business. Early in the process, Rawdin typically engages an investment banker skilled in and familiar with the relevant industry to consider potential buyers for the company.
He also helps the family prepare for a potential sale, drafting a strategic plan and capital budget, preparing clean financial statements and making sure the firm's record-keeping system is in order.
Read MoreFamilies find trusts appealing
He may also, depending on potential buyers, help prepare pro-forma financial statements reflecting possible strategic changes in the business.
The objective, said Rawdin, is to make it look like the family is running a responsible business and to maximize the earnings potential of the enterprise. "When you sell a business, you typically receive a multiple of earnings, so every dollar saved in earnings results in $7 or $8 more on the purchase price," he said.
Rawdin doesn't discount the emotional turmoil involved in selling a family business. For some family members, the business is their career, and it defines who they are. A sale of the company may cause resentments and recriminations between family members that will never be resolved.
Nevertheless, a family in intractable conflict has to recognize that their wealth and legacy are in danger and that a sale of the business may be the only viable option.
Read MoreSuccession tips for small businesses
"It's always tough selling a family business," said Rawdin. "This is their lifeblood, and they don't want to sell their family history. But, " he added, "there's no shame in selling."