Alphonso’s Automated Content Recognition software is now there in 34 million households in the US tracking everything that’s being played on the TV. If you live in the US, there is a 1 in 4 chance that Alphonso is in your home too. But you’ve probably never heard this name before. Intriguing? Read on.

Imagine you’re out with your friends and the topic of mattresses randomly comes up. You pitch in, someone hints that they are looking to buy one, and the conversation swiftly moves on. The next day you’re browsing the internet and bam! mattress ads suddenly start creeping up on every page. These kinds of stories are no longer just big-brother style dystopia fiction. More and more people are experiencing the same and you might have a similar story as well. A popular theory to explain all this is that your phone is stealthily listening to you, so it knows what you’re looking for. But these claims have not been conclusively proven and social media companies vehemently deny it.

A similarly eerie experience is seeing ads on your phone for products related to the shows you are watching on TV. For example, seeing Avenger’s merchandise ads the day after you’ve watched the movie. Or seeing the same TV ads show up on your phone and laptop. You could dismiss it as a coincidence, but it is most likely not. Enter Alphonso.

What is Alphonso?

In Dec 2017, the New York Times published an article claiming that some apps on your phone might be tracking what you’re watching on TV. This was possible by bundling the app with software made by the company Alphonso. Alphonso uses the phone’s microphone whenever the app is open, listening to whatever is being played on the TV. It then analyses this data and provides it to brands who use it to make marketing strategies. In 2017, Alphonso’s CEO Ashish Chordia revealed that over 1000 apps on the Play Store have integrated Alphonso’s Automated Content Recognition (ACR) software. These numbers are much higher now.

Is it legal?

Yes, because Alphonso gets your consent before it starts tracking.

Apps that include Alphonso’s ACR are given a privacy template to include in their T&Cs that clearly states that the microphone will be used with the user’s consent to record audio from the user’s surroundings and this recording will be matched with Alphonso’s database. It also states that this information will be shared with a wide range of advertisers. In addition to this, users have to also explicitly give permission for microphone access and apps have to convey how the microphone will be used. The consent message might be something like this: “This app uses audio to detect TV ads and content and shows appropriate mobile ads.” Apps are also required to provide an easy method to opt-out from this tracking and allow people to use the app without it.

Many of us accept these requests without reading through the description and become victims unknowingly. But these steps are sufficient to comply with the Federal Trade Commissions privacy protection policy, thus making Alphonso’s work legal.

How exactly does Alphonso work?

1. Alphonso’s servers receive live feeds from over 200 broadcast networks across the US. It uses its patented algorithms to processes and analyzes the ads played on these networks frame-by-frame and detects details such as brand, category, product, network, date, time, the TV program it was associated with, etc; creating a unique digital signature for each ad which is stored in a TV index. Alphonso also has such an index for movies and video games.

2. On the client-side, Alphonso works with app developers, smart TV manufacturers, set-top box companies to integrate their automated content recognition (ACR) software in their devices.

3. Through these different sources, Alphonso can collect a trove of data. For example, you’re watching a TV show and the commercial break comes on. Given that a lot of people tend to grab their mobile phones during commercials, you do the same. You open an app, perhaps the game Honey Quest, that has Alphonso’s ACR installed. If you had given this app permission to access your microphone when you installed it, it automatically uses this privilege to record audio from devices around you, which is then processed by the ACR software in the background and matched with the digital signatures in Alphonso’s database. This works even when you close the app or put the phone in your pocket as long as the app is open in the background.

With the data collected, Alphonso now has vast information on which channels and ads are being watched in a household, what kind of programs are popular, what time of the day do people in that household watch TV, the IP address, location, etc. Alphonso can work its magic even without mobile apps by relying on its partnerships with TV and set-top box manufacturers who share viewer data with Alphonso. Big brands like LG, Samsung, Vizio, and Sony are among such partners.

Alphonso also works with other data-mining companies like Experian to pair this data with the demographics based on location and interests based on browsing patterns associated with an IP address.

4. Since Alphonso records audio snippets from all sources around, it can also collect audio from gaming devices, cinema-goers, non-ad serving platforms like HBO GO and Netflix, etc; and match this with their database to get an idea of what kind of TV shows or games are played in a household or what type of movies people go to watch.

5. With all this data, brands have various advertising strategies that they can pursue. If a brand’s ads are being frequently watched on TV, they can retarget the same viewers with internet and mobile ads to reinforce their product. This is particularly effective because viewers are highly likely to grab their mobiles during commercials break. Brands can “conquest”, which is to allow a brand to show the viewer an ad on the phone when a competitor’s ad is playing on TV. If Pizza Hut’s ad is on TV, Domino’s can show their ad on the mobile device.

Brands can also target specific customers based on their viewing preferences. Even if viewers watch shows on non-ad serving platforms, such as Game of Thrones on HBO GO, these viewers can be shown ads for related products like GoT merchandise. They can target video game players with products related to their interests.

Political ads based on demographics and viewers’ network preferences are just a click away. For example, if a household watches the right-leaning Fox News a lot, Republican parties can use this to their advantage by showing donation ads to these viewers and increase their chances of successful fundraising.

Read this post that describes some weirdly fascinating inferences that Alphonso was able to make with the ad data they collected: “MSNBC viewers need to smile more often (Smile Direct Club), Fox News viewers are watching their figures (Nutrisystem, South Beach Diet, and Jenny Craig) and CNN folks must address a serious skin condition (Pharma brands Otelza and Humira).”

6. Finally, Alphonso also provides metrics showing conversions. If a customer watched a particular brand’s ad, it can then use the smartphone’s location to pair this with their visit to the brand’s store. Or it could use data from its partners to show that the brand’s website was visited after they saw an ad. This attribution allows brands to verify which marketing campaigns are working and which aren’t.

With the industry’s largest TV viewership footprint, Alphonso has been serving valuable data to the biggest brands in the world to help strategize their advertising. While Alphonso claims that its software never records human speech or any other personally identifiable information, it is only a matter of time before we come across a company that does. Although these tracking techniques might be legal, whether it is ethical is an open debate.

It’s scary finding out that a company you’ve never heard of before is the reason behind many of the personalized ads you see every day. It is scarier to know that the hardware in your hands or your homes might be recording audio from its surroundings. But I hope after reading this article you’re more cautious about the permissions you give to apps and are better informed of the ads that are being served to you.