1
vote
1answer
416 views

Web Crawling Legal Issues [closed]

I am wondering if there are any legal guidelines concerning web-crawling. Assuming that I respect the robots.txt, what else do I have to take into account. Are there any restrictions regarding e.g. ...
1
vote
3answers
2k views

Is scanning a site directory considered illegal penetration testing?

Does anyone know if using Dirbuster [link] to scan for unlisted directories and their contents is (possibly) legal in say the US?