"Consumers expect immediate gratification," says Lori Schafer, executive adviser at SAS Institute, which creates software for retailers. As a result, she says retailers need to develop technology in weeks, instead of months or years.
Retailers are playing catch-up after several years of watching shoppers gradually move from physical stores to the Web. Online sales have grown from 5.9 percent of $2.64 trillion in total retail sales in 2009 to 7.6 percent of the $3.1 trillion in revenue last year, according to Forrester Research.
The explosion of people using smartphones to shop has pushed stores to move faster. U.S. consumers are now spending more than half of their time on retailers' websites using their smartphones and tablets, according to the National Retail Federation, a retail trade group.
Retailers knew they needed to figure out how to create online and mobile technology, so they began looking to the Silicon Valley where they can tap the talent, culture and creativity that comes from tech giants like Facebook and Apple.
(Read more: Will iPad minis be in short supply this holiday?)
Wal-Mart Stores, the world's largest retailer, was the first to open a tech lab in Silicon Valley. Since opening Wal-MartLabs in San Bruno in 2011, the company has rolled out a number of technologies that it developed there.
One of the biggest projects? Wal-Mart rebuilt its website's search engine, which launched in 2012. It can guess a customer's intent when he or she types a term rather than just returning specific search results. A search for "denim" yields results for "jeans" instead of products with "denim," for example.
Wal-Mart's mobile app also has been a big focus at Wal-MartLabs, which has 1,200 workers and all the trappings of a Silicon Valley startup including treadmill desks and ping pong tables. For instance, Wal-MartLabs developed technology that enables Wal-Mart's mobile app to show products and give discount offers to customers based on the aisle they're in. It also developed technology that enables the mobile app to track customers' spending based on a predetermined budget.
Wal-Mart, which is based in Bentonville, Ark., says having a presence in Silicon Valley has been invaluable in part because it offers the company early access to technology entrepreneurs. For example, two years ago, Wal-MartLabs met the founders of a startup called Grabble as they were in Silicon Valley pitching their technology that enables customers to get receipts for their purchases by email. Wal-Mart has since bought the startup, hired the founders, and next year, shoppers will be able to get the so-called e-receipts.
The company says it's so pleased with its results at Wal-MartLabs that it plans to open another tech office in nearby Sunnyvale in January. It also has smaller tech hubs elsewhere. "We are not a retailer in Silicon Valley," says Neil Ashe, CEO of Wal-Mart's global e-commerce operations. "We are building an Internet technology company inside the largest retailer."
Teen retailer American Eagle Outfitters opened its tech center in San Francisco in July. The 10,000-square-foot location is filled with movable desks to encourage spontaneous brainstorming sessions among its 20 workers. The goal: to get to know more about American Eagle's customers.
Among its projects is an effort to consolidate the personal data of American Eagle customers, including their shopping history, from the company's email campaigns and loyalty programs. The retailer wants to gain information that would help it better target its marketing around a customer's buying habits.
(Read more: Samsung creams Apple in third quarter smartphone sales)
Executives at American Eagle, which is based in Pittsburgh, say that's just the beginning of the types of technology that could come out of its Silicon Valley center. Joe Megibow, a senior vice president and director of American Eagle's tech center, says he envisions a future when customers can go into the store and have the clerk pull up information on a tablet, such as the shopper's email address and buying history.
Target's focus at its 5,000 square foot office in the historic Folgers Coffee Co. building is more futuristic. The lab, which opened with 20 workers in May, is looking at how wearable gadgets like smartwatches— computerized watches that communicate with smartphones—can be used in its stores.
Additionally, Target, which is based in Minneapolis, is experimenting with a mobile app feature that would allow customers to call up information like whether certain cereal is gluten-free by taking a photo of the box with their cellphone camera.
None of the projects at Target's lab have made it to shoppers yet, but David Newman, director of the center, says the goal is to test "the unexplored and underexplored."
—By The Associated Press