Trump also had a heated exchange with Cruz when the senator said Trump would appoint liberals to the Supreme Court.
"You are the biggest liar," Trump said sharply.
As they tried to talk over each other, Cruz chided Trump by saying, "Adults don't interrupt."
Ohio Governor John Kasich, who finished second in the New Hampshire primary last Tuesday and who pushes an optimistic message, called for calm.
"These attacks, some of them are personal. I think we're fixing to lose the election to (Democratic front-runner) Hillary Clinton," he said.
Trump's attacks on the Bush family carried risks for him, since many U.S. military veterans in South Carolina have long supported the family.
Cruz and Rubio renewed their battle over who is the toughest on illegal immigration with Cruz insisting that Rubio, as part of a Gang of Eight senators who sought a compromise on legislation in 2013, was for "amnesty" but now is against it for political purposes.
He insisted that Rubio had said in Spanish on Univision that he would not rescind an executive order signed by President Barack Obama in support of the children of illegal immigrants.
Rubio shot back: "I don't know how he knows what I said on Univision because he doesn't speaking Spanish."
As the crowd roared, Rubio said Cruz is "telling lies... He's lying about all sorts of things and now he makes things up."
The death of Supreme Court Justice Antonin Scalia also loomed large over the debate with over 9,000 mentions during the first half, according to social media analytic firm Brandwatch.
Before the clashes broke out, the Republican candidates urged Obama not to nominate a successor to Scalia, saying it should be up to the next president to decide.
Scalia's death, announced earlier on Saturday, and the consequences for the conservatives' 5-4 advantage on the high court cast a shadow over the ninth debate between rivals for the Republican presidential nomination. [nL2N15S0JE]
The Bush campaign laid into Trump for criticizing his family. "I'm sick and tired of Trump attacking my family. This isn't about my family or his family. It's about your family. #GOPDebate," Bush's official account (@JebBush) tweeted during the debate.
The hashtag #GOPDebate was the top-trending item on Twitter Saturday night. Trump dominated the bulk of the conversation in the first half, with 42 percent of the mentions associated with the hashtag #GOPDebate, according to Twitter data. Bush was a distant second, with 23 percent of the conversation.