He listed the company's continuation of negative comparable-store sales and a dependence on more "trend-right fashion" as reasons for caution.
Read MoreCoach's profit falls as North American comps slide
"Also, in our view, a traffic restoration is tough to expect for any retailer," Chen wrote.
Coach on Tuesday said sales declined 5 percent in the quarter ended March 29, including a dramatic 21 percent same-store sales decrease in North America. It was the fourth-straight quarter of domestic same-store sales declines. The company, known for accessibly priced luxury handbags, said that category continued to be its weak spot, while men's and footwear showed momentum.
Many have pointed to Coach's new creative director, Stuart Vevers, as a potential catalyst for the company, which has lost share to more fashion-forward firms in its category, namely Michael Kors and Kate Spade. With a background at British fashion house Mulberry and LVMH-owned Loewe, the company is betting on his fashion resume to give its products a style boost.