Fast-casual restaurants, which offer quality ingredients prepared quickly, are widely regarded as a bright spot in the ho-hum restaurant category.
Why then are some of the stocks, namely Panera Bread, Noodles & Co. and Potbelly, sinking year-to-date?
The answer lies in sky-high expectations and fierce competition, analysts say.
"Potbelly and Noodles were recent IPOs. If you look at the valuations, they were ridiculous," said Steve West, restaurant analyst at Investment Technology Group. "So when the companies report earnings and sales that don't meet expectations, they'll get butchered on the Street."
Read More Shake Shack preparing for an IPO: Sources
Nicole Miller Regan, senior research analyst at Piper Jaffray, called the former valuations of some recent IPOs "really out of control."
For example, Noodles' price-to-earnings ratio over the next 12 months has fallen to 45 on Friday, from 93 last October, according to Factset data. This same measure dropped for Potbelly to 52 on Friday, from 87 in October.
"They're trading almost where they should be—at a multiple that's more reflective of history, but is still a premium of growth," Regan told CNBC.
Year-to-date, Potbelly shares have dropped nearly 49 percent, while Noodles stock is down about 42 percent.
Despite some individual name stocks falling off a cliff, the segment itself continues to perform well.
U.S. fast-casual sales last year rose 11.3 percent—a multiple of the growth compared with other restaurant segments. In contrast, quick-service restaurants rose just 2.3 percent, while full-service restaurants inched up 2.8 percent, according to food industry research firm Technomic.
"Businesses and stock valuations often diverge," said Nick Setyan, an equity analyst at Wedbush Securities. "You can have a very solid business and very solid growth rates … and expectations can simply diverge from the fundamentals at times. I think this is a very good case of that—expectations being much higher than they should have been."
Read MoreEating alone? The new American diner flies solo
Much of the fast-casual category's growth is coming from the ethnic food category, while competition is especially fierce in the sandwich space, Setyan said during a phone interview. Fast-casual pizzas also are showing strong growth.
Although Panera Bread helped create the fast-casual category, "there's an argument that perhaps Panera's best days are behind and there's not much growth left," Setyan added.
Year-to-date, Panera Bread's stock is off 16 percent as it attempts a tech-based revamp of in-store operations.
The rapidly blurring lines in the industry as many fast-food chains adopt elements of fast-casual models are putting pressure on the fast-casual segment, Regan said.
Read MoreNever wait for a restaurant check again—here's how
The success of Chipotle Mexican Grill, which has reported virtually unparalleled comps lately in the industry, is also putting pressure on competitors. The burrito maker's stock has surged 28 percent this year.
"If Chipotle is knocking it out of the park, someone's got to be suffering," said West of Investment Technology Group in a phone interview.
These strong numbers suggest Chipotle is taking share in a nearly stagnant industry. Despite emerging from a recession, restaurant traffic continues to be weak as many diners haven't returned to their former spending habits. For the year ended in May, traffic contracted 0.3 percent while average check ticked up 2.2 percent, according to market research firm The NPD Group.
"It's the same thing we've been seeing for the last five to six years," West said. "We continue to see pressure on traffic."
Despite the pullback in some fast-casual names, West still sees growth for the space.
"If you look at a long-term perspective, fast casual is where you'd want to be because people are becoming more health conscious," he said.
"I think there's opportunity for some of these concepts long term, but right now they're struggling," he added.