The good times have been rolling recently for watchmaker Fossil Group. But investors should remember how fragile the business can be.
The watch industry fell into trouble during the 2000s, with many people using mobile phones to tell time and deciding simply not to wear a timepiece. Domestic sales of mid-priced watches—those costing $150 to $1,000—fell steadily from $2.5 billion in 2005 to $2.2 billion in 2008, according to research firm Euromonitor International. The decline was especially notable during boom years when most of the retail sector thrived.
But after the financial crisis, the industry experienced a welcome revival. Cash-strapped consumers still wanted brand names, but couldn't spend a small fortune on them. Watchmakers wisely seized on the opportunity to sell timepieces adorned with fashion brands.
Read MoreMichael Kors stock not as haute as before
Perhaps the greatest beneficiary of that trend was Fossil, which began to experience a pickup in sales of Michael Kors watches, available for as little as $120. Fossil likely got an extra boost from Kors because the brand had become incredibly popular as it began rolling out stores and raising awareness.
Fossil's financial performance reveals just how critical Kors has been. Sales of Kors watches and jewelry at Fossil totaled over $300 million in 2011 and soared to $730 million in 2013. Meanwhile, overall sales at the company rose from $2.6 billion to $3.3 billion. That indicates Kors sales accounted for a whopping 60 percent of revenue growth over the two-year period.
The trouble is that Kors probably isn't being compensated sufficiently for its contribution to Fossil's bottom line. The current licensing agreement, which is set to expire at the end of next year, includes a royalty payable to Michael Kors equal to 10 percent of sales of items adorned with its logo, Nomura analyst Simeon Siegel estimates. In contrast, Fossil has other agreements with royalty rates as high as 20 percent, according to company filings. Fossil declined to comment to CNBC.com, and Alecia Pulman of ICR declined to comment on behalf of Kors.
Kors probably has the ability to put some pressure on Fossil. After all, there are alternative watchmakers that Kors could use. Movado, for instance, currently produces watches for Coach under a similar licensing agreement.
Read MoreMichael Kors rally won't set a retail trend
For a watch business that's approaching $1 billion in revenue, Kors simply isn't seeing a great deal of profit. Assuming the royalty payment was 10 percent last year, Fossil would have paid Kors $73 million. That compares with the roughly $1 billion of total operating income Kors earned in the year through March on $3.3 billion in sales.
Of course, it would be disruptive for Kors to halt production with Fossil and try to find another watchmaker. The success of the watch business also serves a marketing function, and Kors even sells some Fossil-made watches in its own stores. That makes it likely the companies will negotiate a renewal.
How much could Fossil get hurt? Suppose the companies increase the royalty rate to, say, 15 percent from 10 percent. If Fossil sells $1.4 billion in Kors merchandise in 2016, Fossil would have to give up $70 million in additional royalty fees. Assuming a tax rate of 30 percent, the impact on net income would be a decline about $50 million. That would indicate a roughly 10 percent hit to consensus 2016 net income estimates. The annual impact would be similar in future years as well.
Fossil's dependence on Kors would be all the greater if the broader fashion-watch craze cools. Corinna Freedman, a retail analyst at Wedbush Securities, said Kors watches have been outperforming the rest of the fashion market, which appears to have plateaued.
Read MoreRadioShack's success story: Not the Super Bowl ad
There's also a question mark over traditional wristwatches as technology companies enter the fray. Samsung and others have already launched smart watches that offer Internet connectivity, and Apple is widely expected to introduce an "iWatch" later this year. Apple, known for its secrecy, hasn't confirmed those plans.
Any degree of success among smart watch makers would be a blow to Fossil. Ike Boruchow, an analyst at Sterne Agee & Leach, estimates that the smart watch market could cannibalize anywhere from 2 percent to 12 percent of Fossil's annual watch sales.
Investors who have stuck with Fossil point to its international potential as reason to accept all these risks. But overseas expansion is itself a risky move in retail. Some major brands including Gap have had to pull out of Asian markets at significant expense when their products didn't resonate. L Brands founder Leslie Wexner has been cautious about expanding overseas, even with a globally coveted brand such as Victoria's Secret.
Fossil's stock is no bargain, trading at 14.3 times 2014 earnings. With so many risks hanging over the company, investors might consider running while there's still time.
—By CNBC's John Jannarone.