Barclays said recently Alibaba could be worth $55 billion.
"He has a very strong reputation and he has a very strong background in the e-commerce space," said analyst Elinor Leung at Hong Kong-based brokerage CLSA. "Ideally, you always want the founder to lead the IPO, but if the founder decides not to do it, Jonathan has a lot of experience."
Lu was hired in 2000 to build out Alibaba's South China sales region. He has also led a team to establish Alibaba's e-payment unit, Alipay, and served as president of the unit. In 2008, he moved to shopping website Taobao and was appointed as its head in 2010.
Both units were successful under his leadership and were key to making Alibaba Group dominant in a country where 16 percent of the population shops online.
In 2011, Lu was chosen by Ma to lead the group's business-to-business e-commerce platform, Alibaba.com, after the unit's chief executive stepped down following a string of fraud cases.
Lu is known to be more low-key than Ma, who worked as an English teacher before building an e-commerce juggernaut. Ma is known for his quips and brash style, and his ease in speaking both English and Mandarin, while Lu is low profile and more comfortable speaking Mandarin.
Billionaire Ma is the 11th richest man in China, according to Forbes, and is one of China's best known corporate leaders. Ma announced his decision to step down as CEO in January, citing his desire to pass baton to a younger generation of leaders. He will remain chairman of the group.
Alibaba is facing competition from e-commerce platforms including 360buy, E-commerce China Dangdang, Amazon.com, and Tencent Holdings.
Last year, Alibaba bought back a partial share of the stake Yahoo owned in it and may be preparing for an IPO. Yahoo still owns 24 percent of Alibaba Group. (CNBC has a content-sharing partnership with Yahoo's finance site.)
(Read More: 3 M&A Targets for Yahoo)
Although Alibaba has said it has no timeline for an IPO, industry sources said the likely Hong Kong listing is among the big IPOs bankers are preparing for late this year or early next year.
In the agreement inked between Yahoo and Alibaba last year, Yahoo built in financial incentives for Alibaba if it lists by 2015.