Rising food prices, harsh weather and a jobs slowdown have all been cited as key factors in a slowing economy. Those factors are taking their toll on the restaurant business, too, chef and restaurateur Bobby Flay told CNBC's "Closing Bell."
"I think consumer spending is OK," he said. "I have some higher-end restaurants and my check average is $60 or $70 a person. Then I have my burger chain, Bobby's Burger Palace. That check averages $10."
Read MoreAs some menus simplify, fast food remains full of additives
While across the board he's seeing foot traffic in his restaurants, Flay admits that labor costs have gone up, and food costs are going through the roof.
"It's a lot harder to make it in the restaurant business," he said. However, he insists that food inflation will not be reflected on his menu.
"You have to take a loss when a loss is given to you," he said. "I don't want to scare the consumer away by raising prices."
Read MoreLast orders: Are hipster brewers up for sale?
—By CNBC's Crystal Lau