Are private equity firms about to become much less private? Or lose an easy way to make tens of millions of dollars? One of the two is likely if an anonymous industry executive's whistle-blower complaint gets traction.
Crain's New York Business has reported that a "senior" private equity insider filed a compliant with the Securities and Exchange Commission earlier this year about the so-called transaction fees that PE firms often charge when they buy a company.
At issue is whether relatively lightly regulated private equity firms can advise on transactions just like an investment bank without registering as a broker-dealer. Such a designation would come with greater regulatory oversight, something PE funds would hate.
The whistler blower, who stands to make up to 30 percent of money recovered by the government, argues that private equity firms are violating federal securities law by awarding themselves multi-million dollar payments for sealing a deal, such as the $40 million Clayton Dubilier & Rice took in 2011 when it closed its $3.2 billion acquisition of Emergency Medical Services.
The private equity industry has been indignant in reaction, mostly because fees are often disclosed during a deal and haggled over with fund investors.
"This is a waste of time," said one private equity firm employee who asked to remain nameless. "The fee is a negotiated deal between very sophisticated parties. No one is being taken advantage of here."
Plus, an increasing number of private equity clients are reimbursed for the transaction fees. Buyout funds launched in 2012 and 2013 rebate an average of 87 percent of transaction fees to their investors, according to industry data tracker Preqin. About 63 percent give the entire fee back to investors, up from 51 percent in 2010 and 2011.
Firms like Warburg Pincus and Hellman & Friedman have traditionally not charged such fees, and other firms like Bain Capital have more recently joined the fee-refund club. The ones that still do, such as TPG Partners and Apollo Global Management, are already being pressured by public pensions to stop the practice.
(Read more: U.S.pension funds pressure Caesars' private equity owners)
For most observers, the whistler-blower charge was hardly a revelation.
"Most limited partners have always hated transaction fees," said Susan Long McAndrews, a partner at $24 billion fund of private equity funds Pantheon. She joked that the practice is sometimes called the "crack cocaine of the private equity industry" because the fees aren't traditionally subject to minimum performance requirements.
While McAndrews was quick to note that PE investors have successfully pushed for more fee rebates, there may be something to the new SEC focus.
"Yes, it's gone on forever and it's not new information. But there could be a case where, by the letter of the law, taking these fees does change the nature of what that firm is doing from being an asset manager to a broker dealer," McAndrews said.
(Read more: Private equity needs to act 'maturely': Report)
Many firms are already watching the SEC for movement on the issue. In April, senior SEC lawyer David Blass gave a speech noting the potential that PE shops might need to be registered as broker-dealers in part because of their transaction fees. Blass called for "dialogue" on the issue.
That dialogue came—mainly in the form of the industry strongly voicing its opposition to the idea.
"Layering broker-dealer regulations on private equity will be of no meaningful benefit to investors and would levy significant costs on private equity firms," said Steve Judge, president of the chief industry lobbyist Private Equity Growth Capital Council.
Judge argues that services provided by PE firms to their affiliated funds and portfolio companies are investment advisory services, a designation that already faces more scrutiny under the Dodd-Frank Act.
Jordan Thomas, a partner at Labaton Sucharow and the whistle-blower's lawyer, disagrees.
"For almost a decade, many of the largest and most prominent private equity firms appear to have chosen to increase their bottom line without subjecting themselves to the oversight, limitations and expenses of the required broker dealer regulatory regime," Thomas said.
He estimates that private equity firms have made more than $2 billion dollars in transaction fees over the last 10 years without being properly registered as broker dealers.
Law firms like Covington & Burling have warned their private equity clients to be prepared for transaction fee questions during so-called "presence exams" for newly registered firms.
"I'm seeing many clients who are apprehensive about taking transaction fees going forward," said Lindsey Simon, founder of regulatory consultancy Simon Compliance.
She said several of her private equity funds clients have stopped charging the fees and are waiting for guidance from the SEC before deciding whether to resume collecting them at all. Simon declined to provide the names of those clients.
The call for greater scrutiny by the SEC comes just as a bill in Congress would exempt most PE firms from registering with the SEC.
—By CNBC's Lawrence Delevingne. Follow him on Twitter