Bad weather bore the brunt of the blame as retailers delivered another underwhelming round of first-quarter results on Wednesday.
But former Bloomingdale's CEO Michael Gould said there's another underlying issue in the sector that's stifling its success: There's not enough risk-taking or newness.
"You're a prisoner of the quarterly report," he said. "The problem in the business today is that risk-taking and fashion is an oxymoron."
Read MoreIts not just Sears: 6 mistakes US stores make
In a competitive environment where retailers should be focused on differentiating their assortments from competitors, Gould said many are instead focused on an exit strategy, such as how they can mark down or return merchandise that doesn't sell well.
He pointed to specialty retailer Urban Outfitters' first-quarter performance as an example of the importance of the assortment, as its on-trend Free People and Anthropologie units posted big comparable-store sales gains, while its struggling namesake division saw a double-digit decline.
Gould and former CEOs from Sears, Toys R Us and Saks weighed in on the current state of retail and strategies for the future.
Former Saks CEO Steve Sadove said the industry is also a mixed bag because of the uneven consumer recovery. Although the higher-end customer is holding up well, as seen in Tiffany's big beat on Wednesday, the lower-income shoppers continue to struggle.
Read MoreWal-Mart's biggest problem: Its customers
He added that the companies that are investing in integrating their online and in-store strategies, such as Saks and Nordstrom, are among those winning.
"There's no question in my mind the consumer wants product anywhere anytime that they want," he said.
Target needs to move faster
Former Sears Canada CEO Mark Cohen said that it could take years for the discounter to recover from missteps into the Canadian market, including higher prices, empty shelves on popular products, and a portfolio of stores in bad locations.
But Cohen said he doesn't think the retailer has any choice but to figure out a way to make the expansion profitable.
Read MoreTarget misses Wall Street estimates
"I don't think they have an escape hatch," he said.
Former Toys R Us CEO Jerry Storch said Target needs to move faster not only in correcting its problems in Canada, but also in improving its online shopping experience, and its merchandise.
"It's almost a visceral reaction like it's not possible that such a great company could have come to this," Stoch said. "Well, the reality is, scrape off the tarnish, there's still a great company under."
"It's going to be one of the most straightforward [companies] to fix and it's going to take a lot of energy and time, but it's going to happen."
—By CNBC's Kristina Gustafson.