Sign up ×
Information Security Stack Exchange is a question and answer site for information security professionals. It's 100% free, no registration required.

I am wondering if there are any legal guidelines concerning web-crawling. Assuming that I respect the robots.txt, what else do I have to take into account. Are there any restrictions regarding e.g.

  • Naming of the Agent
  • Requests per Second

What can happen if I ignore the robots.txt? Is there any sort of central knowledgebase for things like these (maybe even for different countries respectively) where I could get started. I just want to know what might be considered as illegal to avoid problems.

Thank You all very much in advance

share|improve this question

closed as off topic by Lucas Kauffman, Rory Alsop Aug 12 '12 at 18:53

Questions on Information Security Stack Exchange are expected to relate to Information security within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.

This appears to be asking for legal advice - so it is offtopic. That said, I must point out that Andrew's answer below is not true - you may breach the Terms of use of the website, or even worse, if you hit it with too many connections per second and DoS the site, you could potentially be liable. If I were you I would look closely at the terms of use of the website, and take it easy with the connections per second. – Rory Alsop Aug 12 '12 at 18:56

1 Answer 1

No issues. Even 1M connections per sec is OK as long as it's not directed towards single subnet. You need to respect the robots.txt and do this without taking down servers / networks.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.