This giant restaurant segment is shrinking

What's the state of the restaurant industry? It depends where you look.

"Mom and Pop" chains are bearing the brunt of the industry's contraction, dropping in restaurant count by 3 percent, based on data from The NPD Group that compared restaurants open as of the end of March to a year ago.

Chain restaurants are still expanding in unit size, albeit not by much, buoyed by the rise in fast casual restaurants like Chipotle Mexican Grill. Their number increased 1 percent from a year ago, while fast casual chains' units rose by 7 percent.

Overall total restaurant count fell by 1 percent 630,511. Although their count is shrinking, independents still make up roughly 54 percent of total restaurants in the United States.

NPD sees the drop in restaurant count as a reflection of stalled growth over the past several years.

Read MoreMcDonald's is losing fierce fast-food fight

Over the past five years, traffic has dropped 2 percent at independent restaurants and 3 percent at midscale/family dining restaurants.

Restaurants up with morning diners

"It's a tough road for independent restaurants particularly in a down or even 'soft' economic climate," said Greg Starzynski, NPD Foodservice's director of product management, in a release. "Independent operators do not have the resources of a chain to sustain themselves in slower times."

Broadly, restaurants have grappled with some increased costs of doing business this year, including more expensive labor following several minimum wage hikes and food cost inflation in some key items like eggs and beef.