"The exit is tied to the company's internal capabilities and management rather than the state of the Indian economy or the new government; the timing of the announcement is coincidental," Arvind Singhal, chairman of management consultancy Technopak Advisors told CNBC.
"Carrefour has been struggling in India for quite some time – but it's not to do with the government policy or the local environment. Competitors like Reliance, which launched their wholesale businesses after Carrefour are thriving. There are some companies that get it right and some that don't," he said.
Read MoreNew India rail minister seeks foreign investment
Carrefour entered India in 2010, opening its first cash-and -carry store in Delhi before expanding to Jaipur, Meerut, Agra and Bangalore. The Indian government permits 100 percent foreign ownership in wholesale, or cash-and-carry, stores, but requires a local partnership for a retailer that wants to sell to consumers.
Shishir Sinha, senior analyst, Asia Pacific at research firm Frontier Strategy Group, agrees Carrefour's decision is not tied to the economic or political landscape in India, noting that the company tends to exit markets where it is not in a dominant position.
"Carrefour has exited several markets in the past few years, even from countries where there were no regulations stopping them from doing business in the multi-brand-retail format (as is in the case in India)," he said.
Read MoreIndia likely to ease restrictions for foreign online retailers
"In the last 5 years, they have exit some exciting emerging-markets including Malaysia, Indonesia (at one point their second largest business in Asia), Columbia, and now India," he added.
Taking another look at India
While Carrefour joins the likes of other foreign players like Korean steelmaker POSCO andU.S. giant Wal-Mart in scaling back operations in India, Modi's sweeping victory has prompted many other multinationals to take a second look at India.
Read MoreWal-Mart cozies up to China as it shelves India
Last month, CEO of Uniqlo Tadashi Yanai met with Modi, giving rise to hopes that the Japanese clothing chain is preparing to enter the market.
"As a consulting company, we know that international businesses are beginning to dust off some of the plans they made 3-4 years ago, to start having a fresh look. We have one company later this week visiting in India from U.S.," said Singhal.
Singhal said while it would ambitious to expect billions of foreign direct investment in the next few months, these are the first signs of renewed interest in India.
Business leaders will be keenly watching the upcoming budget on Thursday when the government is expected to lay out its roadmap for the economy and the reforms it intends to carry out over the coming years in areas such as foreign direct investment (FDI) and taxation.