Summers sees the potential for a soft back-to-school sales. Combined with a shorter holiday shopping season, he said, weak sales could persist until the end of the year.
Given those concerns and the companies' current valuations, he downgraded both Wal-Mart and Target to neutral.
Not all analysts are ready to give up on the big-box discounters, however. McKeever expects Target shares to move higher from here given its strong position in the U.S. and optimism about Canada.
(Read more: Wal-Mart outlook reveals a tale of two consumers)
Valuations become an issue
In general, consumer discretionary stocks look expensive relative to other sectors, and Bank of America Merrill Lynch strategists suggest investors tread carefully.
"Consumer discretionary has historically rewarded stock-picking more than other sectors, and we suggest that investors focus on several themes within the sector: beneficiaries of home renovation, business spending, higher-end consumption and improving global consumption," the strategists wrote in a note.
One investor suggested focusing on strong brands.
"In this slower growth environment, we do tend to look for specialty retailers that have very strong brands and have been able to continue to drive traffic and comps," said Lori Wachs, co-founder of Cross Ledge Investments. She pointed to companies such as Urban Outfitters and Michael Kors.
Investors may also want to look beyond the big home improvement retailers to smaller names that fit the home remodeling theme but at a cheaper valuation.
Canaccord Genuity's Champine likes Lumber Liquidators. "They have a new store reset that we think will pay off over several years," she said.
—By CNBC's Justin Menza. Follow him on Twitter @JustinMenza.