The home improvement retailer announced Tuesday it is naming Ellison president and CEO, effective July 2. Ellison will take over for Robert A. Niblock, who previously announced his intention to retire.
Lowe's, under pressure from activist investor D.E. Shaw & Co, announced in March its CEO was stepping down.
Ellison is chairman and CEO of J.C. Penney, which he has attempted to steer through a turnaround. Prior to J.C. Penney, Ellison worked more than 12 years at Lowe's rival Home Depot, including serving as executive vice president of U.S. stores.
Shares of J.C. Penney closed down 6 percent Tuesday, while shares of Lowe's were shed 1.9 percent.
"The turnaround program that Ellison put in place at J.C. Penney has partly delivered but is still far from complete. There is now a question mark over how this plan will proceed and, indeed, whether J.C. Penney will remain on the same trajectory," said Neil Saunders, managing director of GlobalData Retail
"Indeed, exiting before his plan is complete is a tacit admission that he may not be able to deliver what investors are looking for."
A spokesperson for J.C. Penney told CNBC that it was just informed by Ellison a couple days ago of the move, and it was not aware when it last reported earnings. It currently has four executives filling in to run day-to-day decisions and is looking for both internal and external candidates.
The highly leveraged department store has struggled to compete within the quickly evolving retail landscape, as consumers shift their shopping online and away from the mall, where many J.C. Penney stores are located. Last year, it closed more than 100 stores.
J.C. Penney also has been squeezed by discounters Walmart and Target, which have improved their clothing selections. Walmart recently forged a relationship with Lord & Taylor as part of its efforts to make its website a destination for apparel. Target, meanwhile, has been focusing on building out its private label apparel brands, including its succesful Cat & Jack children's line.
Under Ellison, J.C. Penney has taken a number of efforts to help steward a turnaround, including a focus on beauty and appliances. In a bid to boost traffic, the retailer has focused on its salon business, hoping hair-dresser loyalty would turn into shopping frequency.
In March, it eliminated 230 positions and announced the departure of executive vice president of Penney's omnichannel business, Mike Amend.
Ellison is not the first CEO to struggle with a J.C. Penney turnaround. Ron Johnson, who joined J.C. Penney from Apple, took the job at the behest of activist Bill Ackman in 2011, but was ousted in 2013.
Lowe's board also named Richard W. Dreiling, a director of Lowe's since 2012, chairman, effective July 2.