How can I identify that my page is requested by robot, but not user's browser?

Question

How can I detect that my page is requested by robot, but not user's browser? I'm aware of basics tricks:

1. Watch for incorrect headers or urls. For example, urls with hash or header with full url - GET www.yoursite.com/test
2. Detect that several unrelated pages were requested directly by some IP (not good enough)
3. Watch for missing headers
4. Watch for obsolete User Agents

Is there any other ways to detect robots?

UPDATE: some good robots identify themselves in User-Agent header:

User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Answer 1

Javascript detection seems to be the common way to detect a bot, by having a random token generated using Javascript and being sent along with requests.

I like the idea of having a 'honeypot' page on your web application. This page would be linked to using a hidden link, which no end user would ever see. This way if this honeypot page is ever reached (by a bot scanning the HTML source for links), you can be pretty confident that this is a bot or other malicious scanner and temporarily block that IP address.

There are also tools out there (such as weblabyrinth) which aim to trap bots in a loop by dynamically generating bogus links for the bot to follow.

Answer 2

Some of the features that are used in web-log mining to identify if a request is made by robot or not:

• click-number: number of http requests in a session
• HTML-to-image ratio: robots generally ignore images
• percentage of pdf/ps requests: robots will request for all of them, whereas humans selectively request a few
• percentage of 4xx error responses: robots will have this metric higher
• percentage of HTTP HEAD requests:to reduce the amount of data from a site, robots use HEAD requests
• percentage of requests with unassigned referrers
• Robots.txt file request
• standard deviation of requested page's depth : high for robots
• percentage of consecutive sequental HTTP requests : higher for human sessions (browser making requests for images/scripts contained in the page on behalf of user)

These are taken from "Feature evaluation for web crawler detection with data mining techniques" http://www.sciencedirect.com/science/article/pii/S0957417412002382

Answer 3

Generally, robots won't execute JavaScript (probably 99% cases), however be aware some robots use WebKit, which runs the JS and renders the complete page.

You could check entries in the logfile. It is likely to be a robot if:

• it requests the HTML page but no images
• first requests a robots.txt
• it's from google etc network
• you can see it's crawling and not browsing

Answer 4

Real solid advice so far.

I just wanted to say that JS challenges are not to be used as sole indicators as some bots (i.e. some Google bots but also some bad/disguised ones) will execute JS and on the other hand some users will not activate JS for security reasons (rare but it happens).

The best thing to do is to cross-examine the bot using the following rules:

1. Method - "POST" is fishy
2. Header Order - if abnormal then it's fishy (but this can be just a proxy thing)
3. robots.txt - if it goes there first, it's very suspicious
4. IP range - if from "unusual" geo-location then it deserves another look
5. Request rate - very high rate is very suspicious
6. Assets - if it does not download files (images, css, js and etc) then it's fishy
7. JS and Cookie challenge - if it cannot pass it, then it's fishy

Setting a score for each test (do not give all the same "value" as some are more important) and combining the data will provide best results.

@xpn-security: it was great to learn about the weblabyrinth. Thanks.