Investors—And Consumers—Are Still Focused on the High End

While the economic data show that a recovery is indeed in place, it doesn't seem as if the message has gotten through to the retailers serving lower- and middle-income consumers.

That may be why some of those gathered at the Piper Jaffray Consumer Conference Wednesday and Thursday at the New York Palace Hotel said they were focused on the high-end market.

Steve Moore, a partner at the private equity firm Brentwood Associates, said his interest lies primarily in companies that cater to higher-income consumers because that group is benefiting most from the surging stock market, recovering housing market and improving employment picture.

The companies that establish emotional brand connections with consumers are poised for the best long-term growth, he said.

The Gap's flagship store in San Francisco
Getty Images
The Gap's flagship store in San Francisco

(Read More: Lululemon CEO's Exit Risks an Apple-Like Scenario)

Michael Kors is a company many look to imitate when it comes to brand equity. The retailer's presentation, given by CFO Joseph Parsons, attracted a lot of interest. The company's revenue jumped 68 percent last year, but he struck a cautious tone about going after shoppers too far outside its high-income core.

"We love outlets, we think it brings in a different customer," Parsons said. "But we want to be careful so we aren't considered an outlet source."

When the conversation turned to international sales, he said Kors expects that Japan will be its highest-gross-margin region long term, but establishing the brand there has to be done meticulously.

"It's critical to be viewed as a luxury source in Japan," Parsons said. "We are being patient."

Gap, however, is still emphasizing the lower end, especially with its Old Navy brand and its Gap outlet locations.

"Forty percent of the $300 billion in North American apparel sales is done in the 'value' sector," said Gap CEO Glenn Murphy.

While the Gap is not truly a "fast-fashion" apparel retailer, he said it expects its truest competitors are global fast-fashion retailers Inditex's Zara, H&M and Fast Retailing's Uniqlo.

"I can't think of anybody else we compete against the competes from value to luxury," Murphy said.

The company has been relatively mum about its plans for its purchase of Intermix, a luxury brand, but in a conversation with CNBC, Murphy said, "We are going to turbo-charge online for Intermix." He also expects to expand the physical footprint to 40 locations from 32 by year-end.

(Read More: Teen Angst: Retailers Fight for Relevance)

Even in the teen panel, higher-end brands were a prominent theme. Fifteen teens from Northern Highlands Regional High School in Allendale, N.J., were questioned by investors in the audience and Piper Jaffray research analyst Erinn Murphy gauging preference ahead of the key back-to-school shopping season.

While favorite brands included Michael Kors, J.Crew, Vera Bradley and Coach, the group seemed quite savvy when it came to navigating all channels for the best value.

One girl said she can find what she's looking for at a clearance price online more easily than in-store.

When we asked the group how they would spend $100 given to them by their parents for back-to-school shopping, preference split when it came to buying one great item at a higher-end store like J.Crew, or a couple of fashion tops at the less expensive Forever 21.

The least-preferred higher-end brand among teens on the panel: Abercrombie. Comments about the teen apparel retailer ranged from: "I don't like it when the name of the store is printed all over the clothes" to "Abercrombie is very immature—that's what middle-schoolers wear, like my little sister and brother."

And it's almost certain that, whether good times and in bad, no one wants to look like a younger sibling.

—By CNBC's Courtney Reagan. Follow her on Twitter @CourtReagan.