I am wondering if there are any legal guidelines concerning web-crawling. Assuming that I respect the robots.txt, what else do I have to take into account. Are there any restrictions regarding e.g.
- Naming of the Agent
- Requests per Second
What can happen if I ignore the robots.txt? Is there any sort of central knowledgebase for things like these (maybe even for different countries respectively) where I could get started. I just want to know what might be considered as illegal to avoid problems.
Thank You all very much in advance