Information Security Stack Exchange is a question and answer site for information security professionals. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I am wondering if there are any legal guidelines concerning web-crawling. Assuming that I respect the robots.txt, what else do I have to take into account. Are there any restrictions regarding e.g.

  • Naming of the Agent
  • Requests per Second

What can happen if I ignore the robots.txt? Is there any sort of central knowledgebase for things like these (maybe even for different countries respectively) where I could get started. I just want to know what might be considered as illegal to avoid problems.

Thank You all very much in advance

share|improve this question

closed as off topic by Lucas Kauffman, Rory Alsop Aug 12 '12 at 18:53

Questions on Information Security Stack Exchange are expected to relate to Information security within the scope defined by the community. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about reopening questions here.If this question can be reworded to fit the rules in the help center, please edit the question.