81

A person talks about a certain thing (product or service) with another person and a short time after the talk the person gets the advertising of the discussed thing on the mobile or desktop device.

I heard and read about such occurrences and didn't know what to think about it. Until some days ago I've personally experienced such occurrence: discussed with my wife a certain product and some days after the talk got advertising of it on Facebook.

My question: is it just an accident and there is not any cause to think about private security issue or are browsers on mobile devices indeed analyze talking through allowed microphone access?

It is true, such issues, if really exist, are very difficult to research, because there is no direct relation between the mention and appearing of the advert. But, if one realizes this sequence of mention and advertising, it is very... alarming?

  • 5
    "Allowed" access is very different from actually accessing. – schroeder Jul 25 at 11:58
  • 36
    People are way easier to advertise to than they think – alexdriedger Jul 26 at 3:55
  • 8
    Personally I think the fact that these advertising companies know what I want without listening to me much, much creepier (and more impressive) than them listening in on my conversations! – Tim Jul 26 at 19:43
  • 3
    @Evgeniy honestly, I’m aware that I could, but I have many other things to do with my time than try to fight against the abilities of the big advertising companies. – Tim Jul 26 at 20:06
  • 10
    Advertisers don't need to listen to you, it they've already told you what to think and talk about. – James_pic Jul 27 at 8:42
102

Listening using the microphone is unlikely

Listening secretly without consent

While listening using the microphone for collecting data would be technically possible, there's a few things against the theory. The unifying factor is that secretly monitoring conversations is considered unethical and would probably even be illegal. Getting caught of such actions would ruin any company's reputation for quite a long time, which makes it less likely.

One would eventually get caught, because:

  • If the device sends the unprocessed audio from the microphone, that would cause notable network traffic.
  • If the device processes the audio with voice recognition, that would cause notable processor activity.
  • So many are reverse engineering both the processes and network protocols.

Is it worth it, when there are better and legal alternatives? Data sources without recording offline conversations are already overwhelming, as explained later.

To back up this reasoning, the network traffic on both Android and iOS weren't comparable to Hey Siri and OK Google on Wandera's experiment, where they systematically played both pet ads and silence to the devices and compared their metrics. (Thanks, TCooper!)

Upon examining the results, we found nothing to suggest our phones are activating the microphone or transferring data in response to sound. The data consumption and battery consumption changes were minimal, and in most cases, there was no change at all.

Sources you have given the permission to listen to you

There are also legal sources of microphone data like Alexa, Siri and Google voice search. These are not spying on you all the time but do use voice recognition – that's just a voice interface that replaces the search bar. Some problems do arise when such a service is activated accidentally.

The closest example of recording everything has been Samsung SmartTVs back in 2015 when their privacy policy transparently stated that:

Please be aware that if your spoken words include personal or other sensitive information, that information will be among the data captured and transmitted to a third party through your use of Voice Recognition.

Although this was mentioned in the privacy policy, it caused so much uproar that today their privacy policy has changed and they only send recordings related to voice commands, just like the others:

Voice information: Recordings of your voice that we make and store on our servers when you enable this function and use voice commands to control a Service, or when you contact our Customer Service team.

The data collected could also be used in ways you might not know if you haven't read the EULA or the privacy policy carefully – nor understood the legalese used.


Alternative explanations

The following mechanisms / phenomena both exist and complement / reinforce each other.

The Internet knows you better than you do

Everyone is constantly tracked while surfing on the Internet. Tracking cookies can identify the person across several sites and make connections. The searches on search engines are saved and connected (whether they are typed of got using voice recognition). Shopping behaviour is carefully analysed connecting both data voluntarily given using loyalty cards and data left behind involuntarily.

Many things can happen to all this data. It can be sold, connected with data from other sources (anonymized or not), and analysed using algorithms. The results can be and are sold again. This enables advertisers to find carefully selected target groups. The relation doesn't even have to be direct like "people buying tools will soon buy construction materials", but the data may reveal much stranger connections. This is explained in detail with many examples e.g. in Hannah Fry's book Hello World: How to be Human in the Age of the Machine (2018).

The bottom line is that there are ways the advertisers can make good, educated guesses on your potential future needs before you do even without listening to you. That's how you really get surprisingly relevant ads.

Confirmation bias

You talk about hundreds of things during the day. Likewise, you probably see hundreds of ads. Most advertisements are completely irrelevant and about topics you haven't talked, so they are easy to dismiss. However, when you occasionally see advertisements on topics you have been discussing, it starts bugging you, leaving suspicions behind.

Every time this happens, you get more and more convinced that someone must be listening to your conversations, and your phone is the first suspect as your closest friend you even take to the toilet.

This is easy to test by taking a vacation from your smartphone – unless it's glued to you. If it doesn't make any difference in how you see advertisement about the topics you are discussing, then it must be something else than the microphone you always carry with you.

| improve this answer | |
  • Comments are not for extended discussion; this conversation has been moved to chat. – Rory Alsop Jul 30 at 13:46
  • 2
    Here's a great example of advertisers knowing you better than you think, and it even goes "all the way back" to 2012: techland.time.com/2012/02/17/… – Conor Mancone Aug 20 at 9:18
  • @ConorMancone: Thanks for the link. I believe this story is one of those that were told in Hannah Fry's book I recommended in this answer. – Esa Jokinen Aug 20 at 14:51
30

I have experimented some time ago on this topic, by choosing a small pool of plausible products, verifying they were not appearing in my ads, and then discussing aloud (orally, not electronically) far and wide a randomly-chosen half of them with trusted friends involved in the same experiment, and carefully not discussing the others.

After some time we also searched for half of them on the Internet and checked the results.

You can run your own tests on this matter, adopting the same protocol. This should help you clear how the matter lies.

My own conclusions so far (TL;DR there is nothing afoot):

The obvious

If you use a search term with Siri or Alexa or whatever, then the search term ends up in Google Ads, Amazon and so on, not at the same speed.

Once you explicitly search for something on the Internet, all bets are off, and the various advertising-enabling companies will share your data with all advertisers (so expect to buy something on Amazon and find it on Facebook, even if it's annoying like hell - I have already bought the thing, for crying out loud!).

The less obvious

This seems to happen (but we have no hard data for a confidence evaluation) even if you are not actively searching for the term, and the Internet-connected listening appliance is just sitting nearby, listening for the activation phrase.

This stands somewhat to reason. The appliance is "thinking": "Is 'Honey' 'Hey Google'? No." "Is ', how about' 'Hey Google'? No." "Is 'a nice cup of Earl Grey' 'Hey Google'? No." -- but to do this, it might well happen that "Earl Grey" makes its way into the corporate servers for tuning and verification of voice recognition, and some other corporate app might sometimes troll the database for leads.

(Normally, unless the utterance matches the activation cue, the sound bite is not transmitted to the corporate servers. It can be in case of "accidental activations", as @Tim noted. But usually manufacturers reserve the right of downloading "selected sound bites" (for example, and I'm pulling this out of my left ear: whatever reaches 90% of the recognition threshold for "Hey, you!" without reaching the 99% required for device activation, say 'Kaiju'. This allows the manufacturer to tune the device so that it has a stronger rejection of words like 'Kaiju').

Also, "listening" and decoding information from unconventional sources or through unconventional means would incur enormous costs for very little advantage, and the very real risk of alienating one's customer base and/or incurring in lawsuits. It seems to me as if it wouldn't be worth the advertiser's while.

Finally, your "propensity" for a given item might depend on some complex demographic. For instance, you talk about buying a Foobaz with a friend of yours (living nearby etc.). Once you have bought your Foobaz, he is in a group of people where Foobaz sales have just gone up by exactly one piece. So his data are sold as "people who are more likely than average to buy a Foobaz".

From your friend's point of view, he talked with you about a Foobaz, and the next day bam!, he's offered one on Google Ads!

That's why in the test it is important to choose an article you never wanted, and never discuss it with anyone (also, beware of prankster friends :-) ).

The not always so obvious

Lots of new product are advertised to me every week. Usually I don't notice them except as a very low-level nuisance, as I'm not remotely thinking about them .

But by mere chance, some product might come by which is trending, and I saw somewhere else and I talked to someone about, or just happened to think about. When this ad comes by, I feel a jolt and notice it, and remember (it is a form of the "Baader-Meinhof effect").

| improve this answer | |
  • 19
    I think it's worth pointing out that the "wake words" for voice-activated devices are checked on-device (which is one reason why they're relatively inflexible, as well as why the devices don't chew your internet bandwidth 24/7). Only if a wake word is identified (correctly or incorrectly) does any audio leave these devices for more intense processing / eavesdropping. This is known about not just because the companies have announced it and researchers have verified it, but also because human audio interpreters working behind the scenes have leaked info about what they hear. – Dave Jul 25 at 23:37
  • 7
    @Dave this is sadly not how this works in all cases: theguardian.com/technology/2019/jul/26/… it's sometimes sent off anyway. The companies just would like you to believe it's always on-device, but it's only most of the time. – E. T. Jul 26 at 15:38
  • 8
    @E.T. That’s accidental activations. The wake-word is always detected locally. – Tim Jul 26 at 19:48
  • 4
    I think the "The less obvious" part is a fallacy. If you discuss a product with a range of friends, chances are extremely high that at least some of them will 'google' for the product. And the advertises 'know' the connection between your friend and you through various methods, e.g. you have used the same Wifi, you have sent them an - unrelated - link with a tracker ID, they have visited your Ebay sale page, you have sent them a link to the product in something as 'innocent' as a messenger, which often process the link (to render it in an adapted form), etc. – Martin Fürholz Jul 27 at 7:29
  • 1
    @E.T. The fact that all my voice activated devices can activate without an internet connection is pretty strong evidence that they locally process the wake word. – Tim Jul 28 at 15:04
11

Someone else around you could have "given the information away".

Example...

You talked about a holiday with your spouse. Your spouse did not mention it, but did some research, using a shared computer for instance, maybe even using your mobile, using their own device with your account somehow logged in, or from the same IP address.

Advertisers now know of some interest (they may think you're the one interest, they may not know precisely, or they might even know your spouse was the one interested!). And now they are advertising you that holiday.

| improve this answer | |
  • indeed, i've supposed something similar (forgotten research, activated Alexa, Siri or Google assistant) while i was hearing reports about this issue from other. In my case i'm 146% sure, nothing of this appeared, not by me, not by my wife. – Evgeniy Jul 26 at 15:37
  • 2
    When my kids have been in the house, suddenly I get advertising for wargaming products and things with unicorns on them. I still find this rather charming. – RedSonja Jul 27 at 5:32
  • I hear some ISPs offer a system whereby any of their customers can sign into any of their other customers' routers (a separate hotspot that it runs for the purpose). I did always wonder if that would wreak havoc with the ad targeting for a given IP address.. – Caius Jard Jul 27 at 5:40
  • 2
    @Caius Jard I know of only one ISP doing (or having done that), and I've never tried nor checked it (I was a customer by I disabled the function). Their claim was any "guest" would not be online with your IP but get a separate one (iow your router, which is 100% ISP provided and is a custom device they have designed) gets 2 IPs, one for you, one for the shared network. I would suspect other ISPs might do similar for legal reasons (router is not going to log every connection, and then you might get sued for some illegal download or so). – user1532080 Jul 27 at 5:55
6

Yes they are listening.

I've been testing it by specifically talking with some people about set subjects that you are not likely to need in a daily life.

For example, we had a talk about army clothes, discussing how different armies dress. It was situational and this is not a topic any of us was ever interested in, like, at all. We also never searched anything in the internet on this topic, but the phones were lying on the table. The next day we got flooded with ads on buying military-looking clothes. They were everywhere and that lasted for a couple of days maybe.

The similar "coincidence" is happening way to often for it to be a coincidence. Army clothes, special professional bakery items, metal alloy factory machines, car details, apartment rent, warehouses rent, and a lot of other things from situational talks that I have with people happen to suddenly flood the ad spaces of the internet on the next day after I have a talk about them with people when my phone is near.

The fun part is that often these topics are so much unrelated to my interests in the daily life because a lot of them are mentioned in the process of playing table top roleplay games. So although these topics get a lot of repetition during the game sessions while we travel in the imaginary worlds, so that the phone hears it and thinks it's important, I never return to them anymore and I never do a search on any of them because they are just a part of a game session. That is just so very much obvious...

So, for example, the talk about army clothes was a talk about totally imaginary armies of imaginary countries in an imaginary world. I was not even interested in army clothes. We were discussing imaginary equipment that we wanted to take in an imaginary expedition and were thinking of buying better clothes for our characters from the local imaginary shop, chosing what suit of alien armor would be better for the next mission.

I don't know who exactly is listening but judging by the amount and placement of the ads that I get everytime we play a game or just talk, the system behind this is massive.

| improve this answer | |
  • 5
    During the process of these discussions, does anyone at the table at any point look up examples or definitions or anything else? I would be very surprised if not. If they are connected to your wi-fi then the search will have come from your IP address, this could potentially explain at least some amount of the targeted ads - though of course it seems a bit of a stretch. – Klaycon Jul 27 at 14:35
  • 1
    @Klaycon I know what you mean, and sometimes finding references for a game is needed. That happens, but mostly no, we are usually not looking up all this in the internet. Most of the time it's even absolutely pointless to look something like that up because you are interested in specifically game-related aspects of the stuff, not the real stuff itself. When using magical rings that give powers to defeat otherwordly evil creatures, why would you search for wedding rings on the internet? When we began noticing what was happening, we checked a lot of times that it was not anyone of us searching. – noncom Jul 27 at 14:50
  • 3
    Your experience is interesting, but without a proper control group of other topics you haven't discussed with others, this is just anecdotal evidence and is susceptible to confirmation bias. And anecdotal evidence is all there really is - no one has ever found any technical evidence to suggest that this type of surveillance is being performed, any many have searched long and hard for it. – Nuclear Wang Jul 27 at 20:58
  • This isn't really a "test". But regardless, do you or your friends strictly play all your games 100% on pen and paper? Usually nowadays, most players seem to use all sorts of apps. Microsoft Office, Google Docs, Instant-Messengers, Chat rooms, email, social media groups, Role-Playing-Games Stack Exchange, and of course game-specific apps meant for things like D&D. Are you certain all your phone's are malware-free? I mean, really, it doesn't even have to be malware. Up until Android 9 (2018), literally any app you download can record microphone audio in the background. – Clay07g Jul 27 at 23:58
  • Answering to both comments: Yes, it could be some apps on our phones, as I said I'm not sure who exactly is listening, but we are not really using phones for the games since that ruins the atmosphere a bit. We write some texts in GoogleDocs for the game, but they do not mention the details of game sessions and would cause other ads as well. Not sure what you mean about the control group of topics that I haven't discussed with others. There's just a tide of ads about words that were used a lot in verbal discussions and were not written down or searched for, occuring pretty often. – noncom Jul 28 at 14:12
5

As the press reported it, Facebook does record mic of messenger : https://newatlas.com/computers/facebook-not-secretly-listening-conversations/

In this article, they speak about messenger app conversation eavesdropped and transmitted to humans in order to check if the AI correctly transcribed it.
The invoked reason is that the AI listen when the user uses the vocal transcription.

The fact that conversation could also be recorded without the user asks for it, is denied. As the fact that they record could be used for ads purpose.

But it is permitted to doubt, as the process imply many actors and the control of each participant is not a Facebook main concern (like we saw it in the past)

| improve this answer | |
4

I doubt it.

If that was the case, tech researchers would have already detected issues and reported them to the press. Thus all of these “smart” devices that can listen are very carefully managed to ensure that when you shout “Alexa…” or “Hey Siri…” then — and only then — do they act on the sounds around you.

Can you imagine if it could be proven that Google, Amazon and Apple and possibly others were indeed listening in on users? And yes, I am aware of the fact that contractors routinely listen into audio passed into smart speakers as part of their debugging and development process but that is not the same as taking that audio and acting on what is heard to deliver targeted content. What kind of a marketing and publicity nightmare that would be… Speaking of marketing…

It is more likely you are talking about something that is very strongly marketed in your region and thus a topic of conversation and thus the ads.

I’ve heard concerns about this before but — from my own professional life and life experience — I know that regardless of what you think about what you talk about casually, chances are you and your friends are far more saturated with media from advertisers to begin with than you think.

For example, I have been looking for some old pieces of furniture hardware for the past few weeks. I have done searches on Google as well as eBay and even Etsy as a part of my research. Do I now see ads for such hardware on my phone when I do other things? Nope.

In contrast, a few years back there was a very big marketing push for the nutritional supplement known as Huel. I did some basic searching for what it was but not very deep. But when I was bullshitting with some co-workers, I joked about Huel and when they themselves went to a news site, suddenly they saw ads for Huel…

And then a few days later I saw a few unopened bags of Huel on someone’s stoop in Brooklyn and was minority flipped out… But then I realized: I live in a major U.S. city, there is clearly some marketing push for this item and thus ads all over the place.

As for the ads my co-workers and I saw, again my co-workers and I are the perfect target market for some gunky “meal as a drink” product. Thus if places like Facebook and Google already know who we are demographically and know we search for tech stuff from time-to-time, then we would be targeted by ads for that product.

At the end of the day we all have freewill and the ability to make our own decisions. But in a media saturated world, we are utterly soaked in targeted advertising that — no matter what you think — enters our mind and influences us.

Devices are not listening to you, but rather the makers of such devices — and the software on them — are deliberately barraging you with messages all the time.

| improve this answer | |
  • 3
    "Can you imagine if it could be proven that Google, Amazon and Apple and possibly others were indeed listening in on users?" They do this all the time and nobody bats an eye. This is just one of many, many articles on the isssue – Douwe Jul 27 at 15:32
  • 2
    @Douwe Will update my answer to address this, but contractors listening in on data to fix and enhance functionality is completely different than companies then harnessing overheard conversations to deliver ads. 100% not the same. – Giacomo1968 Jul 27 at 17:01
  • 1
    The point is, that the whole "listening in to users would ruin a companies reputation" argument sounds reassuring, but is simply and objectively false. So any argument based on that assumption must therefore also be false. By putting "devices are not listening to you" in bold your basically propagating a lie. – Douwe Jul 28 at 8:00
  • And for instance this: "Thus all of these “smart” devices that can listen are very carefully managed to ensure that when you shout “Alexa…” or “Hey Siri…” then — and only then — do they act on the sounds around you." is also objectively false. – Douwe Jul 28 at 8:02
  • Or what if your African grey parrot activated Google & Alexa during your confidential conversation. – Esa Jokinen Jul 28 at 9:03
3

It seems probable.

"Reputable" companies already don't mind overreaching with this for their voice recognition, often sending off recordings of you even when they weren't actually locally recognized as a command so they can analyze it for their own gain like improving speech recognition:

Doesn't seem like much of a stretch that less caring actors would eventually go further and analyze it for advertising. Might be illegal in many places, but will that stop more shady agencies situated somewhere in other countries? Fake support calls supposedly from Microsoft are also illegal, yet a giant business model. As is e-mail spam.

| improve this answer | |
3

I think there's a lot of speculation here, and while I haven't done extensive research myself, this article I found last year seems to be well researched, and has a very plausible explanation. If I can find other's similar that I've read in the past, will add them shortly.

I think what this mostly adds to the current top answer is pointing out the massive streams of data outside simple online collection methods. i.e. retailer databases and even cash purchases if tied to a loyalty card or account (your phone number at the grocery store)

So you may adamantly claim Facebook must have listened in on your private conversation yesterday about a friend’s wedding and then served you an ad for tailored wedding suits because you have not googled anything wedding-related in years. But there are scores of other data points the system has on you to determine what you should see at any given point. Not only does the system know exactly where you are at every moment, it knows who your friends are, what they are interested in, and who you are spending time with. It can track you across all your devices, log call and text metadata on Android phones, and even watch you write something that you end up deleting and never actually send.

The deeply disconcerting implication of all this is that the rich vein of data constantly being gathered can be crunched by an algorithm to essentially predict what you and your friends are talking about, and serve you an ad that is perfectly tailored to your current needs. Even though these Facebook ad algorithms are not nearly perfect (try to pay attention to how often you are served ads that are entirely irrelevant to your interests), the simple fact that they are so eerily correct even some of the time is the real conspiracy here.

https://newatlas.com/computers/facebook-not-secretly-listening-conversations/

They're clear to note times at which the apps do collect microphone data, but for it happening regularly, see the controlled experiment with a silent room and one with audio, monitoring the phone's processor and network usage.

I've also read another article that I can't find now, which outlined Google's(I believe) approach, which is basically creating an AI instance meant to mirror each individual person they want to serve ads to. It can get scary good at not only knowing what you have looked for/are looking for, but even preemptively advertise for say, vacation packages on a soon-to-be-fiances phone days before her significant other proposes, based on the ring searches and purchases on said significant other's phone.

Tl;Dr; Yes they listen sometimes, no it's not when they say they aren't, it's not worth the risk and they have good enough targeting without needing to.

| improve this answer | |
  • This link and this research was already outlined in another answer – schroeder Jul 27 at 18:56
  • 1
    I wouldn't call it outlined, they touch only on the messenger app recording some conversations sometimes, and that there's doubt they don't use it for advertising how they claim to. I'd argue they're entirely different answers with the same link. – TCooper Jul 27 at 21:40
  • I totally agree with @TCooper: the other answer takes just one part of the article and makes opposite conclusions than the author of the article. The data gathered for this article is actually a missing evidence supporting the reasoning in my answer. +1 – Esa Jokinen Jul 29 at 15:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.