That Game on Your Phone May Be Tracking What You’re Watching on TV

At first glance, the gaming apps — with names like “Pool 3D,” “Beer Pong: Trickshot” and “Real Bowling Strike 10 Pin” — seem innocuous. One called “Honey Quest” features Jumbo, an animated bear.

Posted Updated

, New York Times

At first glance, the gaming apps — with names like “Pool 3D,” “Beer Pong: Trickshot” and “Real Bowling Strike 10 Pin” — seem innocuous. One called “Honey Quest” features Jumbo, an animated bear.

Yet these apps, once downloaded onto a smartphone, have the ability to keep tabs on the viewing habits of their users — some of whom may be children — even when the games are not being played.

It is yet another example of how companies, using devices that many people feel they cannot do without, are documenting how audiences in a rapidly changing entertainment landscape are viewing television and commercials.

The apps use software from Alphonso, a startup that collects TV-viewing data for advertisers. Using a smartphone’s microphone, Alphonso’s software can detail what people watch by identifying audio signals in TV ads and shows, sometimes even matching that information with the places people visit and the movies they see. The information can then be used to target ads more precisely and to try to analyze things like which ads prompted a person to go to a car dealership.

More than 250 games that use Alphonso software are available in the Google Play store; some are also available in Apple’s app store.

Some of the tracking is taking place through gaming apps that do not otherwise involve a smartphone’s microphone, including some apps that are geared toward children. The software can also detect sounds even when a phone is in a pocket if the apps are running in the background.

Alphonso said that its software, which does not record human speech, is clearly explained in app descriptions and privacy policies and that the company cannot gain access to users’ microphones and locations unless they agree.

“The consumer is opting in knowingly and can opt out any time,” Ashish Chordia, Alphonso’s chief executive, said, adding that the company’s disclosures comply with Federal Trade Commission guidelines. The company also provides opt-out instructions on its website. Alphonso declined to say how many people it is collecting data from, and Chordia said that he could not disclose the names of the roughly 1,000 games and the messaging and social apps with Alphonso software because a rival was trying to hurt its relationships with developers. (The New York Times identified many of the apps in question by searching “Alphonso automated” and “Alphonso software” in the Google Play store.)

Chordia also said that Alphonso did not approve of its software being used in apps meant for children. But it was, as of earlier this month, integrated in more than a dozen games like “Teeth Fixed” and “Zap Balloons” from KLAP Edutainment in India, which describes itself as “primarily focusing on offering educational games for kids and students.”

Alphonso is one of several young companies using new technologies to enter living rooms in search of fresh information to sell to marketers. For all the talk of digital disruption in the ad world, television still attracts almost $70 billion in annual spending in the United States, and advertisers will gladly pay to amplify and analyze the effectiveness of that spending.

The spread of these technologies, combined with the proliferation of internet-connected TVs and tools that can identify video content through pixels and audio snippets, has resulted in some questionable practices.

Last year, the trade commission issued a warning to a dozen developers who had installed a piece of software known as Silverpush onto apps with the goal of using device microphones to listen for audio signals that humans could not hear to log what they watched on TV. This year, Vizio agreed to pay $2.2 million to settle charges that it was collecting and selling viewing data from millions of internet-connected televisions without the knowledge or consent of the sets’ owners.

Companies gathering such data, especially through games, need to make their business practices clear to consumers “because it’s so inherently unexpected and surprising,” said Justin Brookman, the director of consumer privacy and technology policy at the advocacy group Consumers Union, and a former policy director at the trade commission who worked on the Silverpush case.

“When you see ‘permission for microphone access for ads,’ it may not be clear to a user that, oh, this means it’s going to be listening to what I do all the time to see if I’m watching ‘Monday Night Football,'” Brookman said. “They need to go above and beyond and be careful to make sure consumers know what’s going on.”

Through its software, Alphonso can follow the ads that people see in friends’ homes and elsewhere. The company has also worked with movie studios to figure out theater-viewing habits, Chordia, Alphonso’s chief executive, said. Smartphone apps that are running Alphonso’s software, even if they are not actively in use, can detect movies based on film snippets provided by the studios ahead of time.

“A lot of the folks will go and turn off their phone, but a small portion of people don’t and put it in their pocket,” Chordia said. “In those cases, we are able to pick up in a small sample who is watching the show or the movie.”

Chordia said that Alphonso has a deal with the music-listening app Shazam, which has microphone access on many phones. Alphonso is able to provide the snippets it picks up to Shazam, he said, which can use its own content-recognition technology to identify users and then sell that information to Alphonso.

Shazam, which Apple recently agreed to buy, declined to comment about Alphonso. Founded in 2013, Alphonso initially focused on working with apps to capitalize on ads through so-called second-screen viewing, as people increasingly turned their attention to smartphones and tablets during TV breaks. Now, the company has broadened its focus to gathering troves of viewing data from companies like TiVo and directly from TVs and streaming devices through deals with manufacturers.

The disparate viewing information is tied to IP addresses, which can be matched to characteristics like age, gender, income and more through big data brokers like Experian without using personally identifiable information like names and addresses.

Still, the connection between microphones and ads is a sticky one. Americans are both inviting internet-connected speakers from Amazon and Google into their homes in droves while expressing anxiety that companies are secretly listening to them and then using that information in unsettling ways, like eerily relevant ads. (Facebook has tried, and failed, to quash that theory many times.)

“We have to be really careful as we have more devices capturing more information in living rooms and bedrooms and on the street and in other people’s homes that the public is not blindsided and surprised by things,” said Dave Morgan, the founder and chief executive of Simulmedia, which works with advertisers on targeted TV ads. “It’s not what’s legal. It is what’s not creepy.” Alphonso’s apps and its relationship with Shazam show that there can be a connection between what our phones may hear and the ads that appear on a website or social media feed in the next few hours.

On the other hand, many people have had issues recognizing audio through apps like Shazam if there is too much background noise, so it is not clear how much information Alphonso’s apps can pick up on a daily basis.

“'It’s not normally, I don’t think, going to be expected that an application is going to be listening for what you’re watching,” Brookman said. “But you’re not necessarily expecting your TV to be watching what you’re doing either.”

Copyright 2023 New York Times News Service. All rights reserved.