I use what I believe is the standard method of identifying bad bots that do not respect my
robots.txt (i.e. disallowing crawling the
bad-bots subdirectory, then logging those who still go there).
So far, I've just blocked bad bots based upon their IP-address after they've been identified. However, I am thinking about introducing a more subtle approach for at least some (e.g. spurious offenders that may be false positives). The idea is that the request will be served, but after a delay (e.g. implemented using PHP
sleep()). That way, a (patient) human being that just happens to be allocated an IP-address previously used by a bad bot will see the page, while a bad bot may just drop the request and move on to the next victim.
The question is: What number of seconds should I use in this type of scheme? Ideally, it should be short enough to not be too annoying for humans, while long enough to deter a bad bot from crawling my site.
Alternatively: Is there some sort of method I could use to measure the time a bot trying to access my site wait for response before "giving up" and moving on?
Edit: The comments by Dinu Smădu and Rory Alsop has convinced me that introducing a delay is not a good idea. And Polynominal has suggested that Security.SE may not be the most suitable venue. So I've taken what I've learned here (thanks again, folks!) and posted a revised question at Webmasters.SE.