7

I use what I believe is the standard method of identifying bad bots that do not respect my robots.txt (i.e. disallowing crawling the bad-bots subdirectory, then logging those who still go there).

So far, I've just blocked bad bots based upon their IP-address after they've been identified. However, I am thinking about introducing a more subtle approach for at least some (e.g. spurious offenders that may be false positives). The idea is that the request will be served, but after a delay (e.g. implemented using PHP sleep()). That way, a (patient) human being that just happens to be allocated an IP-address previously used by a bad bot will see the page, while a bad bot may just drop the request and move on to the next victim.

The question is: What number of seconds should I use in this type of scheme? Ideally, it should be short enough to not be too annoying for humans, while long enough to deter a bad bot from crawling my site.

Alternatively: Is there some sort of method I could use to measure the time a bot trying to access my site wait for response before "giving up" and moving on?

Edit: The comments by Dinu Smădu and Rory Alsop has convinced me that introducing a delay is not a good idea. And Polynominal has suggested that Security.SE may not be the most suitable venue. So I've taken what I've learned here (thanks again, folks!) and posted a revised question at Webmasters.SE.

Improve this question
11
  • 4
    This isn't an answer, but I really wouldn't use a delay for this - humans will not be patient. You need to ensure that the site is as fast as possible otherwise you will lose visitors.
    – Rory Alsop ♦
    Jan 6, 2013 at 11:23
  • Have you looked into a solution based on fail2ban (fail2ban.org)?
    – Deer Hunter
    Jan 6, 2013 at 11:47
  • @Deer Hunter, fail2ban assists in detecting and identifying rogues. I am satisfied with my method for identifying rogues. What I try to do is to improve my methods for dealing with them after they've been identified.
    – Free Radical
    Jan 6, 2013 at 12:08
  • 1
    There's a contradiction in here: on the one hand, you say you're satisfied with rogue-finding, on the other hand you worry about false positives. This is a standard Type I/Type II error statistical decision making problem, you only need to specify your loss function. Please remember to include (@RoryAlsop's argument) loss of visitors due to increased latency into the function.
    – Deer Hunter
    Jan 6, 2013 at 12:22
  • This isn't really a security question, it's more about web administration. Probably better off on Webmasters.SE or ServerFault.
    – Polynomial
    Jan 6, 2013 at 12:49

3 Answers 3

Reset to default
12

I don't think that this kind of approach is a good idea. By using the sleep() function on certain requests you are making it easier for an attacker to create Denial of Service. He would just create a lot of request to the resource you are serving with a delay. This way, the number of requests the server can process at a time would be reached and legitimate users would not be able to access the site.

Moreover, you do not have to "punish" the user by offering him a long waiting time. I would look for alternatives if a webpage is slow all the time.

The best protection is not have sensitive information that is publicly accessible. Use an authentication mechanism to protect those resources and allow access only to the users you want.

If you are trying to prevent bots from copying information from your site, maybe CAPTCHA could help. Use a CAPTCHA before displaying certain information.

Improve this answer
1
  • 2
    Your point about a delay making the site more vulnerable to certain types of DoS attacks is a good one. I also appreciate the related point about delays made in a comment by Rory Alsop. Just to make sure: I do not expect a robots.txt arrangement to protect sensitive information. What I try to do is too keep no-good scrapers that steal content and consume bandwidth off the site. Thank you for mentioning CAPTCHA! IMHO, CAPTCHA is a PITA for legitimate users. However only vetting suspected bad bots with a CAPTCHA may be a better solution than exposing the same segment to a delay.
    – Free Radical
    Jan 6, 2013 at 11:57
3

A delay is unlikely to be of actual use in deterring an automated system. Automated systems will wait indefinitely, probably longer than a person would be willing to, so you are more likely to deter actual customers than robots, so it would be bad for business.

Improve this answer
2

A better approach might be to have a redirect on bad bot IP. This redirect could explain the situation and provide a CAPTCHA to remove the IP from the bad bot file.

Improve this answer
2
  • 1
    And now I see you already thought of that after someone suggested a CAPTCHA protecting sensitive portions of your site.
    – AJ Henderson
    Jan 7, 2013 at 14:16
  • Yup. Unfortunately the moderators over at Webmasters.SE don't think this particular angle merits discussion, so that thread was promptly closed. Oh, well ...
    – Free Radical
    Jan 7, 2013 at 14:23

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .