It's "another disappointing day from retail sales on the heels of Nordstrom having really disappointing sales," said JJ Kinahan, chief strategist at TD Ameritrade.
"The one thing I find interesting today again is crude, which looks like it could flirt again with $40. That will probably weigh on the market all day," he said.
Crude oil came off session lows to settle down $1.01, or 2.42 percent, at $40.74 a barrel.
Read MoreWhy the price of oil has got stock traders spooked
The Nasdaq composite underperformed as Apple lost nearly 3 percent to end below its 50-day moving average. The Nasdaq closed below the psychologically key level of 5,000 for the first time since Oct. 22 and traded below for for the first time in intraday trade since Oct. 23. The index is the only major average still positive year-to-date.
The iShares Nasdaq Biotechnology ETF (IBB) was one of the few advancers in Friday's session, ending up 1.1 percent. The ETF still lost 2 percent for the week.
The major indexes pared losses slightly in midday trade before holding lower.
"I think right now investors are really prone to being nervous. There's the specter of the Fed moving in December," said Thomas Lee of Fundstrat Global Advisors.
In a note Friday, Fundstrat downgraded consumer discretionary to neutral in favor of energy. Lee said rising wages would become a headwind for the sector an increasingly competitive retail environment.
October retail sales showed an increase of 0.1 percent, below expectations of a 0.3 percent rise. Retail sales excluding automobiles, gasoline, building materials and food services rose 0.2 percent after an upwardly revised 0.1 percent gain in September.
Nordstrom closed down nearly 15 percent, for its worst day since July 2000. The stock is down 32 percent year-to-date.
The retailer posted earnings after the close Thursday that missed estimates by 15 cents, on revenue that fell below analyst projections as well. The firm cut its 2015 sales and profit forecast, but unlike competitors who pointed to weather and inventory issues, Nordstrom did not specify a reason for the cut.
Before the opening bell, J.C. Penney posted a smaller-than-expected loss in quarterly earnings, on revenue that topped estimates. The company said earlier in the week that comparable-store sales grew by 6.4 percent, while analysts polled by FactSet expected a 5.7 percent increase.
The stock closed down nearly 15.4 percent, up 14.8 percent for the year so far.
"It's a bit surprising they're selling off as hard as they are. What you're seeing is investors recalibrating expectations for the segment," said Efraim Levy, equity analyst at S&P Capital IQ, which is positive on consumer discretionary. "A lot of company multiples have been rising to the higher end of the historical range."
"Now that you're seeing department stores and other retailers not performing so well, consumers are making purchases elsewhere," he said. "There's a revaluation going on in the sector."
Read MoreEarly movers: JCP, FIT, TYC, EPC, YELP, GME, WEN, TWC, LGF & more
"I think the biggest issue today both for the (consumer) companies and stocks is the inventory risk," said Ike Boruchow, managing director of retailing, department stores and specialty softlines at Wells Fargo.
The retail earnings follow mostly disappointing results from Macy's earlier in the week.
"I still think the opportunity within consumer discretionary is more nuanced than buying the sector at large," said Mark Luschini, chief investment strategist at Janney Montgomery Scott.