Is it noticeable when you use a tool to take a copy of a website? - how likely are you to trigger logs and whatnot when you use tools like HTTrack?

Is it actually illegal or 'bad' to do this? - alls you're doing is browsing the website quite quickly, no?


Is it noticeable? Very much so: hundreds or thousands of rapid-fire requests from a single address will stick out in the logs, and may trigger anti-DoS or anti-crawler measures.

Is it bad? Depends on the site, the rate of copying, and many other factors. Rapidly crawling a website hosted on someone's home server could easily overload their connection and make the site unavailable for other people; if it's dynamically-generated site on shared hosting, the same could happen due to CPU overload.

On the legal side, it depends on the site's terms of service and what you intend to do with your copy. If you are simply copying the site for personal offline use and the ToS doesn't explicitly forbid this, it's unlikely that there will be any problems.

  • 2
    For those who actively fight scraping, usually, it just gets you a sand trap ban where you get molasses response or an outright IP ban that may be temporary or permanent. In the old days of email scraping, some website owners would send you to a honeytrap that spewed garbage. And yep, it's all logged... – Fiasco Labs Sep 28 '14 at 22:42
  • Violating the TOS of a website doesn't mean you are violating the law. A TOS is only an information about behaviors which will cause the site administrator to evoke their property rights and ban you from their service. In case of web scraping you are usually not violating the law, unless you cause damage or put the stuff online on another website (which would be a copyright violation). – Philipp Sep 29 '14 at 7:14
  • 1

It depends.

On one hand, every HTTP request should be assumed to leave an entry in the server's logs. What this means, however, is highly context-dependent. For a small website with only a few pages, a full mirror may be indistinguishible from a human clicking on a small number of links in it. A website containing millions of pages, however, necessarily takes millions of HTTP requests to mirror.

Weblog analysing systems usually include mechanisms for plotting request density geographically, and the statistical machinery that does that can trivially detect spikes of many requests coming from a single IP address or network in a relatively short time.

The owner of the website may or may not object, depending on context. Many companies are generally happy when GoogleBot reads through their websites, for example. They may be much less happy when they notice a competitor is doing the same thing. They may also be unhappy when they notice that either the traffic or computing power needed to generate responses costs them more money than they expected to spend on the service.

Depending on what kind of data is served by the server, you may acquire some legal liabilities by coming to possession of it. For example, in EU, the Data Protection Directive (and national laws implementing it) govern handling of personal data.

In general, if you want to mirror or scrape a website in a polite manner, you should start by requesting /robots.txt, parse it, and follow the instructions in it. You should usually also space out the requests to the server so it wouldn't be choking on your traffic. Nowadays, a few seconds of delay between completing one request and starting a new one is generally considered sufficient. The expected delay used to be longer in the early days of Internet, when bandwidths were lower, processing power was more expensive, and webservers sucked worse at balancing their load.

Websites may also contain legal notices or 'terms of use', which you may or may not care about. In some jurisdictions they may be enforceable by civil courts. In some jurisdictions, criminal prosecution may be a risk if you don't obey them. In some jurisdictions, courts may not care at all. Ask your lawyer. Obviously, a broad-spectrum bot nowadays is generally not expected to be able to understand legalese, just robots.txt, but if you're scraping a particular website, taking a look at the terms may be a good point of preparation.

Sometimes, the terms of use may include reasonable requests by the humans behind the website; if so, you should usually respect their wishes.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.