Last year, comparable same-store sales dropped 25 percent as J.C. Penney customers turned away from its new everyday low price strategy, which replaced heavy discounting and couponing.
As sales have slid, so has the value of both the company's stock price and Ackman's Pershing Square Capital Management fund's stake in the company. The fund currently holds 17.8 percent of the retailer's outstanding shares.
The fund manager helped recruit Johnson last year, and had previously defended him throughout the company's struggles but turned more critical of Johnson on Friday, shortly before he was fired.
On Monday, Ullman laid out some of his top priorities for J.C. Penney in one of his first conversations with Wall Street analysts.
"The biggest thing that he talked about—number 1 priority—is reconnecting with the customer," said Matthew Boss, an analyst at JPMorgan Chase. "He talked about having the right product, but more importantly having the price and the value perception, something he believes was lost over the past year."
—By CNBC.com's Katie Little; Follow on Twitter
Additional News: Saving JC Penney: New CEO Ullman Plans Coupons, Discounts
Additional Views: Why There's Still Hope for J.C. Penney
Ackman's Pershing Square Capital Management fund holds 17.8 percent of the retailer's outstanding shares