There need to be big changes at The Children's Place and it starts with shaking up the board, according to Macellum Capital Management CEO Jonathan Duskin.

Duskin is one of two activist investors attacking the retailer's executive team over performance issues relating to inventory, store closings and merchandising — as well as the stock price and CEO pay.

"We think this is a company which can more than double earnings," said Duskin on Tuesday's "Fast Money."

The Children's Place has been lagging chief competitor Carter's. Over the past three years, the retailer is up about 47% while Carter's is up 100%.

Duskin's firm, along with Barington Capital, has a 2.1% stake in The Children's Place. They're proposing retail veterans Seth Johnson and Robert Mettler for the company's board to help boost the business. Shareholders will vote on the measure this Friday.

The disagreement has led to a great deal of invective being hurled between management and the shareholders.

"They have resorted to mudslinging. It's clear they don't have much of a case left," said Duskin. "They're trying to not let dissidents be heard. They're putting up obstacles and we believe they are misleading investors."

The Children's Place declined an invitation to appear on the show.

In a statement to "Fast Money," a company spokesperson wrote "The Children's Place board and management team have delivered on all fronts, our robust quarterly results, capital return program and share price performance prove that. We believe we are just beginning to see the impact of our transformation plan."