Tell me more ×
IT Security Stack Exchange is a question and answer site for IT security professionals. It's 100% free, no registration required.
up vote 1 down vote favorite

I am wondering if there are any legal guidelines concerning web-crawling. Assuming that I respect the robots.txt, what else do I have to take into account. Are there any restrictions regarding e.g.

  • Naming of the Agent
  • Requests per Second

What can happen if I ignore the robots.txt? Is there any sort of central knowledgebase for things like these (maybe even for different countries respectively) where I could get started. I just want to know what might be considered as illegal to avoid problems.

Thank You all very much in advance

share|improve this question
This appears to be asking for legal advice - so it is offtopic. That said, I must point out that Andrew's answer below is not true - you may breach the Terms of use of the website, or even worse, if you hit it with too many connections per second and DoS the site, you could potentially be liable. If I were you I would look closely at the terms of use of the website, and take it easy with the connections per second. – Rory Alsop♦ Aug 12 '12 at 18:56

closed as off topic by Lucas Kauffman, Rory Alsop♦ Aug 12 '12 at 18:53

Questions on IT Security Stack Exchange are expected to relate to IT security within the scope defined in the FAQ. Consider editing the question or leaving comments for improvement if you believe the question can be reworded to fit within the scope. Read more about closed questions here.

1 Answer

up vote -4 down vote

No issues. Even 1M connections per sec is OK as long as it's not directed towards single subnet. You need to respect the robots.txt and do this without taking down servers / networks.

share|improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.