Alibaba's main Chinese competitor in mobile

Alibaba has now become something of a household name in the U.S. after its largest-ever IPO last month, but back in China it's effort to expand into mobile—the fastest growing part of e-commerce—faces challenges.

The reason: Rival Tencent has a commanding lead in mobile over Alibaba. That growing market is key for any tech company hoping to get a grip on consumers, especially in China.

Tencent's voice and text WeChat is nearly ubiquitous on Chinese smartphones and is growing internationally with 438 million users, up from 300 million last January.

Read MoreWeChat helps Apple rack up bonus points in China

The service enables users to do things like buy emoticons, pay for game apps, invest their money in a fund, hail a taxi and buy goods. Individuals can also set up stores on the platform, just as they can create merchant accounts on Alibaba's Taobao or Tmall e-commerce sites.

"I like [Tencent] for the WeChat dominance," JG Capital analyst Henry Guo said, who is bullish on the future of the firm.

Since its official launch in 2011, WeChat (Weixin in Chinese) has even taken away from the clout of Sina Weibo, China's version of Twitter.

"There's clearly a move away from Weibo to WeChat for security reasons," Forrester analyst Kelland Willis said. She referenced close government monitoring of viral Weibo accounts and the many fake Weibo accounts aimed at generating large followings.

Read MoreChina tightens censorship: No sweat for WeChat

In addition to WeChat, Tencent's QQ instant messaging service and ecosystem are everywhere in China, with 521 million active mobile users as of the second quarter of 2014. Although they lead Chinese mobile e-commerce, Alibaba's mobile shopping apps only generated about 146 million monthly active users in August, according to Chinese research firm EnfoDesk (cited here by China Internet Watch).

However, a spokesman for Alibaba said the company was "very much a mobile company" but was a commerce not a social service.

"Chat users have zero commercial intent," the spokesman added.

The Alibaba spokesman added that it had 188 million active users of its mobile commerce apps in June 2014 and had generated US$71 billion in mobile gross merchandise value (GMV) in the year to June 2014. In the three months ended June, one third of Alibaba's GMV came from mobile.

"Tencent's focus is on social networking. That market gives them an edge in mobile versus traditional e-commerce," Gartner analyst Andrew Frank said. "WeChat is a big revenue generator for Tencent, because it allows in-app purchases."

What's more, some of Alibaba's competitors in retail are working with Tencent. This March, Tencent partnered with JD.com, a Chinese website that sells consumer electronics. Three months later, Tencent also took a 20 percent stake in China's version of Craigslist, 58.com. Both of these companies went public in the United States within the last year and have held above their opening prices.

In the new online-to-offline space, in which a consumer can purchase a product online and pick it up in a store immediately, Tencent joined Chinese search engine Baidu and property conglomerate Dalian Wanda to form Wanda E-commerce at the end of August. Alibaba has a similar deal with another mall developer, Intime.

None of this competition means Alibaba is going anywhere, though.

Forrester analyst Xiaofeng Wang said that Tencent needs to "figure out a better strategy for mobile commerce," beyond these deals, and Willis added that Alibaba is still a "behemoth that no one is going to take away from."

For one thing, Alibaba is catching up in mobile and actually leads in the small number of e-commerce apps in that area, according to EnfoDesk. With strong gains in mobile transactions in the last quarter and the purchase of a local navigation service in China, the company's ambitions in hand-held devices are likely to succeed eventually, said Wang

That said, so far Alibaba's direct attempts to match Tencent's WeChat app have failed. CEO Jack Ma had announced last fall that he was quitting WeChat for Alibaba's similar Laiwang app, and told all his employees to do the same and add 100 friends. However, the most recent user numbers, from last November, show only 10 million users.

Beyond messaging apps and e-commerce

Founded by 42-year-old Ma Huateng, also known as Pony Ma, in 1998, Shenzhen-based Tencent is best known as a gaming and social media leader in China. The firm went public on the Hong Kong Stock Exchange in 2004, while Alibaba's attempts to hold an IPO there failed last year due to a partnership structure controversy. Both Tencent and Alibaba are incorporated in the Cayman Islands.

As a developer of many popular domestic games, Tencent also became the global leader in gaming revenue with about $5.3 billion in 2013, jumping from the previous year's fifth-place standing behind industry leaders such as Activision Blizzard and Electronic Arts, research firm Newzoo reported in March.

However, Tencent has only recently increased its international marketing efforts by working with local advertisers for WeChat in Southeast Asia, Africa and other countries.

Read MoreHow to make a lot more money on Alibaba

"Globally, Alibaba has better brand awareness, and its IPO helped a lot," Guo said.

The battle between the two Chinese tech firms is different from those of Google or Facebook in the United States, because diversified holdings in Web, gaming, mobile, taxi services, retail property, video and more mean the Chinese companies have become large media conglomerates.

Despite these vast properties, Wang pointed out that Alibaba and Tencent are vying for the attention of one limited group: consumers. And increasingly, the best way to reach them is through their hand-held devices.

"In the long term," Guo said, "I believe these companies will compete in mobile head-to-head."

Clarification: This article has been updated since first published to include comment from Alibaba.