I am wondering if there are any legal guidelines concerning web-crawling. Assuming that I respect the robots.txt, what else do I have to take into account. Are there any restrictions regarding e.g. ...
Does anyone know if using Dirbuster [link] to scan for unlisted directories and their contents is (possibly) legal in say the US?