Two federal law enforcement officials, who spoke to The Associated Press on condition of anonymity because they were not authorized to discuss the case, said investigators never believed the brothers were unaware of the fraud and both were the focus of active investigations until the time of their deaths.
The officials acknowledged, however, that being able to substantiate their suspicions proved difficult, and as investigators unraveled the case's complex web of financial chicanery they had increasingly turned their attention to a case they felt they could prove: tax evasion.
Both officials told the AP it was likely Andrew Madoff would have faced tax evasion charges if he had not died. The ultimate goal, they said, was using federal charges as leverage to get him to return money to investors.
Read MoreMadoffson's death adds complication for victims
Attorney Martin Flumenbaum, the executor of Andrew Madoff's will, has maintained that the brothers never "knew of, or knowingly participated, in their father's criminal conduct." Flumenbaum did not respond to a request for comment Friday on the possibility of tax charges against Andrew Madoff.
During a 2011 "60 Minutes" interview, Andrew Madoff said that from the beginning he had "absolutely nothing to hide and I've been eager, almost desperate, to speak out publicly and tell people I am not involved."
Bernard Madoff, now 76, is serving a 150-year prison sentence in a North Carolina prison after admitting he fleeced thousands of investors in a scheme that went on for decades.
Five former high-level Madoff firm employees were convicted earlier this year of helping carry out the fraud by conspiring to defraud clients and falsifying records. It is unclear whether anyone else will be charged, though prosecutors describe the investigation as ongoing and have obtained in the last week delays in the sentencing of several cooperators.
Days after Andrew Madoff's death, his will was made public, detailing an estate worth $16 million. He left his wife Deborah West—who filed for divorce after the fraud was made public—about $5 million, divided his tangible personal property equally among his two college-age daughters and provided his fiancee, Catherine Hooper, with $50,000 a month until the money he left is gone.
Read MoreAndrewMadoff's $16M fortune enrages father's victims
Irving Picard, the trustee who has so far recovered more than half of the $20 billion invested in the Ponzi scheme, is seeking to recover money from the estates of both brothers and has the power to pursue money and assets wherever they are disbursed.
He charged in his recent lawsuit that the Madoff sons knew about the fraud and tried to cover it up by deleting emails during a Securities and Exchange Commission probe.