3

A recent article on scraping tiktok and Facebook states:

On the one hand running selenium headlessly is perfect to keep your machine “cool”, however it may help get you flagged as a scraper. System administrators can spot a headless request with ease.

The author uses a random User-Agent for each request, so I'm not sure where the logs would indicate that a headless version of a browser making the request. Are there any specific signatures to detect a headless browser?

Good faith effort on the question:

With foresight, it looks like one could check make a check for things like webdriver version (which can be spoofed too). However, by stating sys admins, the quote seems to imply that the logs themselves are sufficient to detect a headless browser.

Improve this question
5
  • 2
    In that second article you linked to, there are a lot of precautions that the Medium article is not taking. Perhaps the author of the Medium article was not aware of all the workarounds to avoid detection?
    – Fire Quacker
    Nov 20, 2019 at 21:12
  • @FireQuacker perhaps. I never even considered that there were other methods beyond the User-Agent until I came across this.
    – Hooked
    Nov 20, 2019 at 21:57
  • I'm not sure if "With ease" is accurate, but there are a variety of fingerprinting techniques, any of which may give away a headless browser. I'm not going for a full answer here though because I'm only aware of some general principles, and can't delve into anything with specifics or certainty.
    – Conor Mancone
    Nov 20, 2019 at 22:08
  • Indeed, chrome still has issues with some fingerprinting techniques giving away that a user is in an incognito window, so it wouldn't surprise me at all if some clever checks could detect a headless browser.
    – Conor Mancone
    Nov 20, 2019 at 22:11
  • @ConorMancone It's kind of a tricky situation. Because JavaScript lets you do weird things, most of the simple checks can be worked around. So while there probably is a way to detect using a technique similar to that incognito detector, it will take a little digging to find it. My point is, I'm not really sure that it's "easy" for a system administrator to detect it. More likely, they'd have to do research and switch techniques every 6 months.
    – Fire Quacker
    Nov 20, 2019 at 22:36

1 Answer 1

Reset to default
3

TL;DR "System administrators can spot a headless request with ease": NO. They can surely spot some headless requests with ease. But in general, that's not true.

Quite obviously, it all depends on how the requests are done (how much effort to disguise them) and how the logs are checked and what countermeasures adopted (how much effort goes into penetrating the disguise).

The fact is that an automated (headless or not) browser and a human can, and often do, perform requests in different ways, with different patterns. When they do, this difference can be spotted more or less easily.

The difference is often "intentional", i.e., it derives from a design choice - typically not executing some scripts or not downloading (some) images to reduce traffic, seeing and following transparent links or hidden parts of the DOM. Navigation patterns are clearly influenced by a human's response times. Also, patience and scope of the navigation might be inhuman by need - if you need to scrape a whole website, you won't use a "human like" navigation even if you could. And if you see someone browsing sequentially the whole site, you might not care for it even if you know it's a human.

A browser could also be analyzed actively, e.g. by intercepting mouse events and relaying them via AJAX; even if a headless browser correctly interpreted the Javascript code and was able to perform, in theory, those requests, it would never by itself understand that a human would perforce generate those events, or know how to credibly do the same.

But should a browser (headless or not) be controlled by a mouse and keyboard-like interface by an AI designed to interpret a screen rendering through OCR and scripting, and scrape a site like a human would, it would be almost impossible - and almost certainly intolerably rife with false positives - to tell it apart from a real human being.

Improve this answer
1
  • I think the point of the author was that, without means to disguise it, using the --headless argument in Chrome will be easy to detect. What you're talking about goes beyond the scope of the context of the question.
    – Steve Sether
    Nov 21, 2019 at 2:51

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .