How can I detect that my page is requested by robot, but not user's browser? I'm aware of basics tricks:

  1. Watch for incorrect headers or urls. For example, urls with hash or header with full url - GET www.yoursite.com/test
  2. Detect that several unrelated pages were requested directly by some IP (not good enough)
  3. Watch for missing headers
  4. Watch for obsolete User Agents

Is there any other ways to detect robots?

UPDATE: some good robots identify themselves in User-Agent header:

User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
  • 4
    My gut says you probably shouldnt care (if its a robot) and should rather look for misbehaving signs. But i am sure there is a spam list online. I think this question is more suited to serverfault – user5575 Jul 11 '12 at 20:01
  • 1
    I'll add that you may defeat friendly webcrawlers (like googles) by trying to detect bots. You should write figure out what you don't want to block, what grey area do you not care about and what you actually want to prevent. A grey area could be allowing user uses a tool to download images or bin files automatically with a browser extension. Would you allow that or not? – user5575 Jul 11 '12 at 22:53
  • Which robots are you worrying about? GoogleBot family, or a more nasty family? – curiousguy Jul 16 '12 at 3:51
  • 1
    of interest "Detecting 'stealth' web-crawlers": stackoverflow.com/questions/233192/… – Jacco Jul 16 '12 at 10:21

Javascript detection seems to be the common way to detect a bot, by having a random token generated using Javascript and being sent along with requests.

I like the idea of having a 'honeypot' page on your web application. This page would be linked to using a hidden link, which no end user would ever see. This way if this honeypot page is ever reached (by a bot scanning the HTML source for links), you can be pretty confident that this is a bot or other malicious scanner and temporarily block that IP address.

There are also tools out there (such as weblabyrinth) which aim to trap bots in a loop by dynamically generating bogus links for the bot to follow.

  • 1
    I believe googlebot is smart enough to do not crawl hidden links (otherwise it will be a hole for SEO promoters) – Paul Podlipensky Jul 12 '12 at 0:08
  • When we say hidden links, do we mean links on foreground color = background color ? – CodeExpress Jul 13 '12 at 20:19
  • It is possible to do this using CSS or Javascript to hide the link. If the bot processes the raw HTML looking for links, other than processing the CSS and JavaScript, it is difficult to know if this is a valid link or not. On the other hand, genuine users would never see this link to click on it. – xpn-security Jul 13 '12 at 20:22
  • 2
    @xpn-security "On the other hand, genuine users would never see this link to click on it." Some users sometimes turn off the web page style sheet, esp. when it has ugly colours, poor contrast, when it makes the text tiny, or when it puts most text in a narrow column to make room for unless side columns (sometimes for ads)! I also browse with JS off when possible. – curiousguy Jul 16 '12 at 3:45
  • 1
    @PaulPodlipensky Even worse: Google may punish websites with such hidden links, unless they are nofollow or forbidden that at robots.txt level. – curiousguy Jul 16 '12 at 3:46

Some of the features that are used in web-log mining to identify if a request is made by robot or not:

  • click-number: number of http requests in a session
  • HTML-to-image ratio: robots generally ignore images
  • percentage of pdf/ps requests: robots will request for all of them, whereas humans selectively request a few
  • percentage of 4xx error responses: robots will have this metric higher
  • percentage of HTTP HEAD requests:to reduce the amount of data from a site, robots use HEAD requests
  • percentage of requests with unassigned referrers
  • Robots.txt file request
  • standard deviation of requested page's depth : high for robots
  • percentage of consecutive sequental HTTP requests : higher for human sessions (browser making requests for images/scripts contained in the page on behalf of user)

These are taken from "Feature evaluation for web crawler detection with data mining techniques" http://www.sciencedirect.com/science/article/pii/S0957417412002382


Generally, robots won't execute JavaScript (probably 99% cases), however be aware some robots use WebKit, which runs the JS and renders the complete page.

You could check entries in the logfile. It is likely to be a robot if:

  • it requests the HTML page but no images
  • first requests a robots.txt
  • it's from google etc network
  • you can see it's crawling and not browsing
  • The image suggestion is a nice one. – user5575 Jul 11 '12 at 22:51
  • @acidzombie24 The image suggestion would identify me as a "robot" if I browse your site at work (images disabled by default) – Null Nov 13 '12 at 21:16

Real solid advice so far.

I just wanted to say that JS challenges are not to be used as sole indicators as some bots (i.e. some Google bots but also some bad/disguised ones) will execute JS and on the other hand some users will not activate JS for security reasons (rare but it happens).

The best thing to do is to cross-examine the bot using the following rules:

  1. Method - "POST" is fishy
  2. Header Order - if abnormal then it's fishy (but this can be just a proxy thing)
  3. robots.txt - if it goes there first, it's very suspicious
  4. IP range - if from "unusual" geo-location then it deserves another look
  5. Request rate - very high rate is very suspicious
  6. Assets - if it does not download files (images, css, js and etc) then it's fishy
  7. JS and Cookie challenge - if it cannot pass it, then it's fishy

Setting a score for each test (do not give all the same "value" as some are more important) and combining the data will provide best results.

@xpn-security: it was great to learn about the weblabyrinth. Thanks.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.