Whitney V withdrew its offer to buy Herbalife after its board rejected Whitney's buyout offer of $38 a share.
Herbalife shares fell 3.5%, or $1.38, to $38.84 on the news.
In a letter to Herbalife, disclosed in a filing with the U.S. Securities and Exchange Commission, Whitney said it was disappointed at Herbalife's unwillingness to discuss the proposal.
Whitney said it would continue to monitor developments in the company and may engage in sale or purchase of the company's common shares. Whitney owned about 27 percent of the Herbalife shares as of Feb. 2, when it made its offer.
Herbalife, which sells weight-loss and nutritional supplements, had on Thursday said Whitney's offer was too low, but added it remained open to a higher bid from Whitney.
In a research note, Morgan Stanley analyst William Pecoriello said Whitney's refusal to improve the offer came as a surprise, adding Herbalife's fundamentals are stronger than when Whitney made its offer.
The analyst expects a negative reaction to the news, but viewed it as an attractive opportunity for fundamental investors, and reiterated his "overweight" rating on the stock.