Do advertisers listen as we talk?

Question

A person talks about a certain thing (product or service) with another person and a short time after the talk the person gets the advertising of the discussed thing on the mobile or desktop device.

I heard and read about such occurrences and didn't know what to think about it. Until some days ago I've personally experienced such occurrence: discussed with my wife a certain product and some days after the talk got advertising of it on Facebook.

My question: is it just an accident and there is not any cause to think about private security issue or are browsers on mobile devices indeed analyze talking through allowed microphone access?

It is true, such issues, if really exist, are very difficult to research, because there is no direct relation between the mention and appearing of the advert. But, if one realizes this sequence of mention and advertising, it is very... alarming?

Answer 1

Listening using the microphone is unlikely

Listening secretly without consent

While listening using the microphone for collecting data would be technically possible, there's a few things against the theory. The unifying factor is that secretly monitoring conversations is considered unethical and would probably even be illegal. Getting caught of such actions would ruin any company's reputation for quite a long time, which makes it less likely.

One would eventually get caught, because:

• If the device sends the unprocessed audio from the microphone, that would cause notable network traffic.
• If the device processes the audio with voice recognition, that would cause notable processor activity.
• So many are reverse engineering both the processes and network protocols.

Is it worth it, when there are better and legal alternatives? Data sources without recording offline conversations are already overwhelming, as explained later.

To back up this reasoning, the network traffic on both Android and iOS weren't comparable to Hey Siri and OK Google on Wandera's experiment, where they systematically played both pet ads and silence to the devices and compared their metrics. (Thanks, TCooper!)

Upon examining the results, we found nothing to suggest our phones are activating the microphone or transferring data in response to sound. The data consumption and battery consumption changes were minimal, and in most cases, there was no change at all.

Sources you have given the permission to listen to you

There are also legal sources of microphone data like Alexa, Siri and Google voice search. These are not spying on you all the time but do use voice recognition – that's just a voice interface that replaces the search bar. Some problems do arise when such a service is activated accidentally.

The closest example of recording everything has been Samsung SmartTVs back in 2015 when their privacy policy transparently stated that:

Please be aware that if your spoken words include personal or other sensitive information, that information will be among the data captured and transmitted to a third party through your use of Voice Recognition.

Although this was mentioned in the privacy policy, it caused so much uproar that today their privacy policy has changed and they only send recordings related to voice commands, just like the others:

Voice information: Recordings of your voice that we make and store on our servers when you enable this function and use voice commands to control a Service, or when you contact our Customer Service team.

The data collected could also be used in ways you might not know if you haven't read the EULA or the privacy policy carefully – nor understood the legalese used.

---

Alternative explanations

The following mechanisms / phenomena both exist and complement / reinforce each other.

The Internet knows you better than you do

Everyone is constantly tracked while surfing on the Internet. Tracking cookies can identify the person across several sites and make connections. The searches on search engines are saved and connected (whether they are typed of got using voice recognition). Shopping behaviour is carefully analysed connecting both data voluntarily given using loyalty cards and data left behind involuntarily.

Many things can happen to all this data. It can be sold, connected with data from other sources (anonymized or not), and analysed using algorithms. The results can be and are sold again. This enables advertisers to find carefully selected target groups. The relation doesn't even have to be direct like "people buying tools will soon buy construction materials", but the data may reveal much stranger connections. This is explained in detail with many examples e.g. in Hannah Fry's book Hello World: How to be Human in the Age of the Machine (2018).

The bottom line is that there are ways the advertisers can make good, educated guesses on your potential future needs before you do even without listening to you. That's how you really get surprisingly relevant ads.

Confirmation bias

You talk about hundreds of things during the day. Likewise, you probably see hundreds of ads. Most advertisements are completely irrelevant and about topics you haven't talked, so they are easy to dismiss. However, when you occasionally see advertisements on topics you have been discussing, it starts bugging you, leaving suspicions behind.

Every time this happens, you get more and more convinced that someone must be listening to your conversations, and your phone is the first suspect as your closest friend you even take to the toilet.

This is easy to test by taking a vacation from your smartphone – unless it's glued to you. If it doesn't make any difference in how you see advertisement about the topics you are discussing, then it must be something else than the microphone you always carry with you.

Answer 2

I have experimented some time ago on this topic, by choosing a small pool of plausible products, verifying they were not appearing in my ads, and then discussing aloud (orally, not electronically) far and wide a randomly-chosen half of them with trusted friends involved in the same experiment, and carefully not discussing the others.

After some time we also searched for half of them on the Internet and checked the results.

You can run your own tests on this matter, adopting the same protocol. This should help you clear how the matter lies.

My own conclusions so far (TL;DR there is nothing afoot):

The obvious

If you use a search term with Siri or Alexa or whatever, then the search term ends up in Google Ads, Amazon and so on, not at the same speed.

Once you explicitly search for something on the Internet, all bets are off, and the various advertising-enabling companies will share your data with all advertisers (so expect to buy something on Amazon and find it on Facebook, even if it's annoying like hell - I have already bought the thing, for crying out loud!).

The less obvious

This seems to happen (but we have no hard data for a confidence evaluation) even if you are not actively searching for the term, and the Internet-connected listening appliance is just sitting nearby, listening for the activation phrase.

This stands somewhat to reason. The appliance is "thinking": "Is 'Honey' 'Hey Google'? No." "Is ', how about' 'Hey Google'? No." "Is 'a nice cup of Earl Grey' 'Hey Google'? No." -- but to do this, it might well happen that "Earl Grey" makes its way into the corporate servers for tuning and verification of voice recognition, and some other corporate app might sometimes troll the database for leads.

(Normally, unless the utterance matches the activation cue, the sound bite is not transmitted to the corporate servers. It can be in case of "accidental activations", as @Tim noted. But usually manufacturers reserve the right of downloading "selected sound bites" (for example, and I'm pulling this out of my left ear: whatever reaches 90% of the recognition threshold for "Hey, you!" without reaching the 99% required for device activation, say 'Kaiju'. This allows the manufacturer to tune the device so that it has a stronger rejection of words like 'Kaiju').

Also, "listening" and decoding information from unconventional sources or through unconventional means would incur enormous costs for very little advantage, and the very real risk of alienating one's customer base and/or incurring in lawsuits. It seems to me as if it wouldn't be worth the advertiser's while.

Finally, your "propensity" for a given item might depend on some complex demographic. For instance, you talk about buying a Foobaz with a friend of yours (living nearby etc.). Once you have bought your Foobaz, he is in a group of people where Foobaz sales have just gone up by exactly one piece. So his data are sold as "people who are more likely than average to buy a Foobaz".

From your friend's point of view, he talked with you about a Foobaz, and the next day bam!, he's offered one on Google Ads!

That's why in the test it is important to choose an article you never wanted, and never discuss it with anyone (also, beware of prankster friends :-) ).

The not always so obvious

Lots of new product are advertised to me every week. Usually I don't notice them except as a very low-level nuisance, as I'm not remotely thinking about them .

But by mere chance, some product might come by which is trending, and I saw somewhere else and I talked to someone about, or just happened to think about. When this ad comes by, I feel a jolt and notice it, and remember (it is a form of the "Baader-Meinhof effect").

Answer 3

Someone else around you could have "given the information away".

Example...

You talked about a holiday with your spouse. Your spouse did not mention it, but did some research, using a shared computer for instance, maybe even using your mobile, using their own device with your account somehow logged in, or from the same IP address.

Advertisers now know of some interest (they may think you're the one interest, they may not know precisely, or they might even know your spouse was the one interested!). And now they are advertising you that holiday.

Answer 4

As the press reported it, Facebook does record mic of messenger : https://newatlas.com/computers/facebook-not-secretly-listening-conversations/

In this article, they speak about messenger app conversation eavesdropped and transmitted to humans in order to check if the AI correctly transcribed it.
The invoked reason is that the AI listen when the user uses the vocal transcription.

The fact that conversation could also be recorded without the user asks for it, is denied. As the fact that they record could be used for ads purpose.

But it is permitted to doubt, as the process imply many actors and the control of each participant is not a Facebook main concern (like we saw it in the past)

Answer 5

Yes they are listening.

I've been testing it by specifically talking with some people about set subjects that you are not likely to need in a daily life.

For example, we had a talk about army clothes, discussing how different armies dress. It was situational and this is not a topic any of us was ever interested in, like, at all. We also never searched anything in the internet on this topic, but the phones were lying on the table. The next day we got flooded with ads on buying military-looking clothes. They were everywhere and that lasted for a couple of days maybe.

The similar "coincidence" is happening way to often for it to be a coincidence. Army clothes, special professional bakery items, metal alloy factory machines, car details, apartment rent, warehouses rent, and a lot of other things from situational talks that I have with people happen to suddenly flood the ad spaces of the internet on the next day after I have a talk about them with people when my phone is near.

The fun part is that often these topics are so much unrelated to my interests in the daily life because a lot of them are mentioned in the process of playing table top roleplay games. So although these topics get a lot of repetition during the game sessions while we travel in the imaginary worlds, so that the phone hears it and thinks it's important, I never return to them anymore and I never do a search on any of them because they are just a part of a game session. That is just so very much obvious...

So, for example, the talk about army clothes was a talk about totally imaginary armies of imaginary countries in an imaginary world. I was not even interested in army clothes. We were discussing imaginary equipment that we wanted to take in an imaginary expedition and were thinking of buying better clothes for our characters from the local imaginary shop, chosing what suit of alien armor would be better for the next mission.

I don't know who exactly is listening but judging by the amount and placement of the ads that I get everytime we play a game or just talk, the system behind this is massive.

Answer 6

I doubt it.

If that was the case, tech researchers would have already detected issues and reported them to the press. Thus all of these “smart” devices that can listen are very carefully managed to ensure that when you shout “Alexa…” or “Hey Siri…” then — and only then — do they act on the sounds around you.

Can you imagine if it could be proven that Google, Amazon and Apple and possibly others were indeed listening in on users? And yes, I am aware of the fact that contractors routinely listen into audio passed into smart speakers as part of their debugging and development process but that is not the same as taking that audio and acting on what is heard to deliver targeted content. What kind of a marketing and publicity nightmare that would be… Speaking of marketing…

It is more likely you are talking about something that is very strongly marketed in your region and thus a topic of conversation and thus the ads.

I’ve heard concerns about this before but — from my own professional life and life experience — I know that regardless of what you think about what you talk about casually, chances are you and your friends are far more saturated with media from advertisers to begin with than you think.

For example, I have been looking for some old pieces of furniture hardware for the past few weeks. I have done searches on Google as well as eBay and even Etsy as a part of my research. Do I now see ads for such hardware on my phone when I do other things? Nope.

In contrast, a few years back there was a very big marketing push for the nutritional supplement known as Huel. I did some basic searching for what it was but not very deep. But when I was bullshitting with some co-workers, I joked about Huel and when they themselves went to a news site, suddenly they saw ads for Huel…

And then a few days later I saw a few unopened bags of Huel on someone’s stoop in Brooklyn and was minority flipped out… But then I realized: I live in a major U.S. city, there is clearly some marketing push for this item and thus ads all over the place.

As for the ads my co-workers and I saw, again my co-workers and I are the perfect target market for some gunky “meal as a drink” product. Thus if places like Facebook and Google already know who we are demographically and know we search for tech stuff from time-to-time, then we would be targeted by ads for that product.

At the end of the day we all have freewill and the ability to make our own decisions. But in a media saturated world, we are utterly soaked in targeted advertising that — no matter what you think — enters our mind and influences us.

Devices are not listening to you, but rather the makers of such devices — and the software on them — are deliberately barraging you with messages all the time.

Answer 7

It seems probable.

"Reputable" companies already don't mind overreaching with this for their voice recognition, often sending off recordings of you even when they weren't actually locally recognized as a command so they can analyze it for their own gain like improving speech recognition:

• David Goldman (CNN): Your Samsung TV is eavesdropping on your private conversations

• The Guardian: Apple contractors 'regularly hear confidential details' on Siri recordings

Doesn't seem like much of a stretch that less caring actors would eventually go further and analyze it for advertising. Might be illegal in many places, but will that stop more shady agencies situated somewhere in other countries? Fake support calls supposedly from Microsoft are also illegal, yet a giant business model. As is e-mail spam.

Answer 8

I think there's a lot of speculation here, and while I haven't done extensive research myself, this article I found last year seems to be well researched, and has a very plausible explanation. If I can find other's similar that I've read in the past, will add them shortly.

I think what this mostly adds to the current top answer is pointing out the massive streams of data outside simple online collection methods. i.e. retailer databases and even cash purchases if tied to a loyalty card or account (your phone number at the grocery store)

So you may adamantly claim Facebook must have listened in on your private conversation yesterday about a friend’s wedding and then served you an ad for tailored wedding suits because you have not googled anything wedding-related in years. But there are scores of other data points the system has on you to determine what you should see at any given point. Not only does the system know exactly where you are at every moment, it knows who your friends are, what they are interested in, and who you are spending time with. It can track you across all your devices, log call and text metadata on Android phones, and even watch you write something that you end up deleting and never actually send.

The deeply disconcerting implication of all this is that the rich vein of data constantly being gathered can be crunched by an algorithm to essentially predict what you and your friends are talking about, and serve you an ad that is perfectly tailored to your current needs. Even though these Facebook ad algorithms are not nearly perfect (try to pay attention to how often you are served ads that are entirely irrelevant to your interests), the simple fact that they are so eerily correct even some of the time is the real conspiracy here.

https://newatlas.com/computers/facebook-not-secretly-listening-conversations/

They're clear to note times at which the apps do collect microphone data, but for it happening regularly, see the controlled experiment with a silent room and one with audio, monitoring the phone's processor and network usage.

I've also read another article that I can't find now, which outlined Google's(I believe) approach, which is basically creating an AI instance meant to mirror each individual person they want to serve ads to. It can get scary good at not only knowing what you have looked for/are looking for, but even preemptively advertise for say, vacation packages on a soon-to-be-fiances phone days before her significant other proposes, based on the ring searches and purchases on said significant other's phone.

Tl;Dr; Yes they listen sometimes, no it's not when they say they aren't, it's not worth the risk and they have good enough targeting without needing to.