4

If there are some chances to detect a bot by considering the users page viewing time, is it possible to measure this time?

When a user request for a page (page request), browser request for several other components within that page (inline requests), i.e. images, CSSs, JSs, ...

By just sniffing the traffic, is it possible to find the time interval between user's page requests (page viewing time) by discarding inline requests? There is a need to differentiate page request and inline requests. Is it possible?

|improve this question
  • 1
    I just left a comment for you regarding a similar question, then realized that you asked that question as well. One way to do anomaly detection that I know works is by using appropriate probability distributions and queueing theory for modeling normal inter-arrival times as a benchmark for comparison. I don't know if it is possible to sniff traffic this finely, to detect anomalous activity though. Well, not if it is practical to do so, in any functional way. I am very curious to read what responses there might be! – Ellie Kesselman Jan 13 '13 at 15:11
5

Here at Incapsula we actually use some of these methods for bot detection (i.e. JS and Cookie challenges) but I`m not sure about the page-view-time metric.

For one thing, you want to detect and block malicious bots as quickly as possible and there are better things you can do then to wait around, even if it's just for a short while.

I also have to agree (+1) with @yfeldblum about the Expire headers and their ability to interrupt resource-load based tests. Having said that, CND and Multiple servers are not an issue if the challenges are performed by the CDN provider himself.

I also wanted to contribute few other ideas for bot detection:

  • Behavior (i.e. early access to robots.txt)
  • HTTP Header order/missing parameters

Also, I would like to suggest not to rely exclusivity on any of these methods because this will lead to false positive identification. Instead, try to combine them into a step-by-step cross verification process. (i.e. JS challenge -if failed-> Cookie challenge -if failed->Robots.txt access and so on....)

Finally, at some point you'll have to start collecting and relying on signatures to minimize redundancy.

After all, you can't keep testing Baidu bot every time it visits, this is just one step away from a self-inflicted DDoS. :)

|improve this answer
4

There are a few problems with this approach:

  1. Multiple Web Servers: if you have multiple web servers, then you must maintain this tracking elsewhere, possibly in a distributed fashion.
  2. CDN Asset Hosting: if you host assets on a CDN, then you would have to extend this tracking to include the CDN-hosted assets.
  3. Asset Fingerprinting and Caching: if you fingerprint your built assets' filenames with the md5 digests of their contents, and set effective Expires: never headers for the fingerprinted assets, then browsers will not request assets from your servers or your CDN most of the time, leading to a very large false-positive rate.
|improve this answer
3

Since HTTP is a stateless protocol you should develop some session level attributes for differentiating between bot and client web sessions. What i recommend is to deploy an application layer firewall that allows to create and maintained session level variables against every web session. Following is a ModSecurity Rule for monitoring web session.

SecRule REQUEST_COOKIES:PHPSESSID !^$ chain,nolog,pass
SecAction setsid:%{REQUEST_COOKIES.PHPSESSID}
SecRule REQUEST_URI "^/cgi-bin/finger$" "pass,log,setvar:session.score=+10"
SecRule SESSION:SCORE "@gt 50" "pass,log,setvar:session.blocked=1"
SecRule SESSION:BLOCKED "@eq 1" "log,deny,status:403"

I would also recommond some tweaking into OpenNMS HTTP monitor it may be of some help.

|improve this answer
  • 1
    Are you recommending a new approach? How setting a cookie would help to differentiate bot from a normal user? Could you please explain it more. – Yasser Jan 13 '13 at 17:40
  • 1
    Cookies is used here as a session identifier for every session. Then for each session separate counters are maintained as Score and Blocked in the example rule.These counters can be increase and decreased based on rule designer logic. – Ali Ahmad Jan 13 '13 at 17:59
3

It sounds like if implemented your approach would catch a lot of bots. There are bots out there that run a full web-stack like webkit or mozilla which would still request the inline resources, set cookies and even run javascript. Sadly, if someone wants to specifically target your site, they could use imacros to write a browser bot that defeats your bot detection approach.

Still, I think its a worthy endeavor so you should go for it.

|improve this answer
2

There are a bunch of other heuristics that I find catches a lot of bots and scrapers:

  1. Manually updating a user agent regex (amazing how many bots actually have words like bot,scraper,agent,spider in the user agent string.
  2. Does this IP create a fresh session on every request (implies that it doesn't store and return session cookies)?
  3. Ratio of IPs per user agent
  4. Ratio of hits per IP
  5. Ratio of types of pages requested (1 index page, 50 item pages looks more like mechanical agent)
  6. Count requests for same page (some bots have bugs)
  7. # page requests per minute (very unlikely that a person will request 100 pages in five minutes.
  8. Geolocate and whois the source IP for IPs that request more than N pages a day (it's unlikely that my English language nyc centric site has a user in Kazakhstan or with an ec2 IP address)
|improve this answer
  • This is not answering OP's question. While your list is something to look into for everyone writing WAF or similar filtering, I believe OP has already considered these (quite possibly implemented too) and is specifically asking for a particular advanced detection method that you don't discuss - detection through requests (or lack thereof) of linked contents. – TildalWave May 15 '13 at 21:07
  • @TildalWave - hmm, this is a tricky one, while I agree that it doesn't answer the asker's question directly, I also think the asker's question (as asked) is completely off topic for Sec.SE. (It's a programing question.) The offer of simpler detections and methods of detection is security related and useful information even if not a direct answer, but does offer an alternative. I guess put another way, the question is completely off topic and this answer is the most on-topic portion of the entire thing even if not an answer to the specific question, but an alternative. – AJ Henderson May 16 '13 at 13:33
  • 1
    @AJHenderson - Heh good observations here, I agree. And as a suggestion to the answerer, maybe it wouldn't be a bad idea to ask a new question and include this answer there, too? It's a nice list and could serve as a reference to others. – TildalWave May 16 '13 at 15:12
1

These two articles (1st and 2nd) are quite related with your question. These are not free so for the ones who doesnt have access to those, let me briefly tell what these does.

By just looking at a hosts activity in the webpage, the authors try to differenciate spambot from human being. In the first article they look at Mouse clicks and movements, Keyboard actions Form field focus and un-focus Form load and submission Page navigation

For your question, if a human fill a textbox in 10 sec on the average, and a person fills that in one sec, this is suspicious and may be a spambot activity. You might use this to suspect but you can't guarantee, fast users are spambots.

There is one important thing here, spambots and crawlers behaves similar, so it is really hard (maybe impossible) to differenciate these two.

|improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.