To figure out the best Super Bowl ad of the year, Omnicom Group's Annalect data platform decided to ask random members of the public. Instead of talking to them, however, the marketing and communication agency used facial recognition technology to figure out which ads they really liked.
The ads that made the most people smile were from Mountain Dew, Marmot and Amazon Echo, while the commercials that generated the highest amounts of "happiness" were from Wix.com, Snickers and . Its results were drastically different from the most viewed ads on YouTube: Pokemon and two spots from T-Mobile.
"We were surprised by the insights," said Anna Nicanorova, the director of Annalect Labs, which helped run the study. "Sometimes the social media activity doesn't necessarily reflect how people react."
When it comes to judging public opinion on a movie or an ad, most companies rely on written or verbal surveys. or social media chatter. Thanks to advances in facial recognition technology and biometric sensors, like Apple Watches, companies now have insights into what exactly tugs at our heartstrings or fills us with rage.
With that knowledge, brands can create ads that not only find people at the right time and right place but in the right emotional state, said Slavi Samardzija, chief analytics officer at Omnicom Media Group's Annalect.
"We are in the business of changing consumer behaviors," he said. "If you think about a consumer and you think of all of that technology, it's kind of like a left brain and right brain. We can take all our data tools and utilize them. All these thingsimpact the rational side and the emotional side of decision making."
Emotion-based data collecting company Realeyes has been using facial recognition software to gauge opinions on advertising. With the participant's permission, it uses webcams and smartphone cameras to "watch" people as they watch ads. Realeyes CEO Mihkel Jaatma estimates that it's run over 7,000 video surveys globally for clients, which include market research company Ipsos and marketing company Mediacom.
"It's the next generation of the survey," said Jaatma. "We are still asking people people via email for some feedback. Rather than 20 minutes, you can just watch a bunch of clips. You just click on a link to the video. There's no user feedback. People prefer that to a tedious question and answer."
Most of the companies using biometric data for marketing use it for research purposes for future ad campaigns. However, because of the speed of data transmission, there is the possibility to use the information in live advertising — and some firms like Lightwave are already doing that.
"Human excitement and human emotion can be used to fuel human experiences," said Lightwave CEO Rana June.
Lightwave uses people's biometric data like heart rates, body temperature and activity levels to create emotion-based advertising. Trackers can range from highly sensitive, company-developed devices to using personally owned smartphones and smartwatches.
At the 2014 South by Southwest Festival, Lightwave worked with Pepsi to host a "bioreactive" concert. Attendees were given wristbands that tracked their movement, temperature and excitement levels. Certain data benchmarks unlocked rewards like visual effects and free Pepsi bottles.
Last year, the company worked with media and marketing agency Mindshare and gave out Apple Watches at the Cannes Lions International Festival of Creativity. There was one caveat: Getting a free device allowed the companies to track the wearer's personal biometric data. At the end of three days, the company hosted a presentation based on a visualization of the information. I
n addition, it worked with Jaguar at Wimbledon in June. Using biometric wristband sensors and air sensors that are able to collect data when people walk by, it judged the crowd reaction and then created an "emotional picture" every 30 minutes that was displayed throughout screens on U.K. trains.
Lightwave was also tapped by 20th Century Fox in January to track select audiences who watched Oscar contender "The Revenant." Unlike typical screenings that rely on written surveys or verbal interviews to gauge audience reaction, moviegoers were given one-time use wristband sensors to gather their biometric data while watching the film. During the duration of the movie, 15 fight or flight responses were measured, heart rate was significantly increased 14 times and the audience was startled nine times.
Lightwave's June explained that in the future, if the data is collected early enough in the movie making process it can use the information to figure out how to edit films or cut movie trailers based on emotional reaction.
"Not only can the data be used for people's emotional experiences, but you can take that same data and create marketing aspects," June said.
There is a bit of a creepiness factor, especially since most of these tactics rely on using personal devices like smartphones or fitness trackers. However, the companies are adamant that they clearly ask for user participation before tracking their data. For the most part, the information is collected as a group so it's impossible for brands to identify one individual.
"We believe this is a combination of arts and science," said Omnicom's Samardzija. "We refer to this as a content inspiration area because of our philosophical beliefs. We don't think this is content dictation. ... Large scale behavioral data, that should inspire creativity. If something is creepy, it's certainly a bad piece of content, and it shouldn't go in the marketplace."
But June said consumers are directly asking them to give them more personalized data because they are curious about their emotional states — in spite of the fact her company is trying to protect them by keeping the information anonymous.
"When you're secretly gathering data, people really don't like that," she said. "If you are explaining a value, you are really demonstrating the why. That's really the question that you have to ask as a marketer: Is your thing a compelling enough reason to be OK with somebody collecting (personal) data?"