Is it possible to protect my site from HTTrack Website Copier or any similar program?
Without setting a max number of HTTP request from users.
Is it possible to protect my site from HTTrack Website Copier or any similar program?
95Easy solution: Just take the website down.– CodesInChaosJul 5, 2013 at 9:57
3HT Track respects exclusions in robots.txt, but "any similar program" might not.– TRiGJul 5, 2013 at 16:28
43But wait, aren't you grabbing other people's websites? stackoverflow.com/questions/17440236/…– GaffJul 5, 2013 at 19:50
13This kind of question is so common, yet so absurd. If you put a website online, when people "view" it, they are downloading it to their computer, you are essentially giving it to them. Then the question becomes "I gave my website to someone, how can I not give it to them?".– laurentJul 6, 2013 at 6:21
4You make the entire site out of some form of server-side code (that way the only things the user would be able to download is what is sent to them).– PixelArtDragonJul 7, 2013 at 0:46
No, there's no way to do it. Without setting connection parameter limits, there's even no way to make it relatively difficult. If a legitimate user can access your website, they can copy its contents, and if they can do it normally with a browser, then they can script it.
You might setup User-Agent restrictions, cookie validation, maximum connections, and many other techniques, but none will stop somebody determined to copy your website.
21You could also watermark all of your images so you could later prove they were originally yours. However that won't prevent the copying it will only help after the fact. Jul 5, 2013 at 14:17
15And don't forget to mention any such measure is more likely annoy legitimate visitors than to hinder someone dedicated to actively circumvent them. Jul 6, 2013 at 14:06
Then how Themeforest manages this? Dec 29, 2015 at 12:23
@Four_0h_Three Watermarks are largely pointless. ANY reasonable watermark can be removed in PhotoShop in 30 seconds (or less). If your watermark can't be removed that easily, then it's probably in an obnoxious location and is completely ruining any enjoyment your users would have gotten from the content. Fact: there is no way from getting viewable hosted content ripped off by someone that really wants it. Accept that fact, and continue on with your life.– MikeMar 24, 2016 at 15:19
Basically, if there's any way to prevent it, there's a way around it. Jun 27, 2016 at 17:00
Protect the part of the site you want to protect with a username and password. Then only assign a username and password to people who sign an NDA, (or similar) that says they won't extract or copy information from your site.
Even then, nothing will prevent a determined skilled attacker from getting an offline copy, you can just make it harder so it's not worthwhile.
9To be fair, it need not be AJAX. Any dynamically generated content regardless of the underlying technology will work in same way - the attacker could easily copy a snapshot of its output only, while the backend (application, database,...) involved in generating these contents is not supposed to be accessible to an unauthorized actor by any other means. Jul 5, 2013 at 11:44
2"NDA, (or similar) that says they won't extract or copy information from your site." - This is laughable. How are you going to enforce such a contract? Will your users tolerate it?– FreiheitJul 5, 2013 at 18:11
3@Freiheit The OP doesn't say if he has a public site or a small B2B site for professionals. If the audience is small, he can target them with identification, etc. How do you enforce it? There are paid services that scan for copyright theft on other sites. Also steganography can be used to track violators on a per-username or IP basis. Jul 5, 2013 at 18:56
As has @Adnan already pointed out in his answer, there is really no way of stopping a determined person from copying snapshots of your website. I used the word snapshots here, because that's what such content scrapers (or harvesters) are really copying. They don't (or at least shouldn't) have access to your backend where your website contents are actually generated and displayed to the end user, so the best they can do is copy its output, one that you can generate in such a way to change in time or adjust according to its intended recipient (DRM schemes, watermarking,...), as has @makerofthings7 pointed out in his answer.
So this much about what's already been answered. But there is one thing about this threat that I feel haven't yet been well covered in mentioned answer. Namely, most of such content scraping is done by opportunistic and automated web crawlers, and we see targeted attacks a lot rarer. Well, at least in numbers - bear with me.
These automated crawlers can actually be blacklisted quite effectively through the use of various WAFs (some might even use honeypots to determine the threats in heuristic ways) that keep updated database of blacklisted domains (CBLs or Community Ban Lists, DBLs or Domain Block Lists, DNSBLs or DNS-based Blackhole Lists,...) where such automated content scrapers are operating from. These WAFs will deny or grant access to your content serving web application based on three main approaches:
Deterministic blacklisting: These are detections based on characteristics of web requests that content scrapers will make. Some of them are: Request originating IP address, Reverse DNS resolved remote hostname, Forward-confirmed reverse DNS lookup (see explanation in one of my questions here), User agent string, Request URL (your web application could for example hide a honeytrap URL address that a content scraper might follow in one of its responses, after it determines the request didn't come from a whitelisted address such as legitimate search engine crawlers / spiders)... and other fingerprint information associated with automated web requests.
Heuristic blacklisting: This is a way to determine a threat either by weighting parameters of a single web request described in the deterministic approach (anti-spam filters use a similar approach based on calculating Bayesian probability), or by analyzing multiple web requests, such as: Request rate, Request order, Number of illegal requests,... that might help determine, if the request comes from a real and intended user, or some automated crawler.
External DNSBL/CBL/DBLs: I've already mentioned relying on external DNSBL/CBL/DBLs (e.g. Project Honey Pot, Spamhaus, UCEPROTECT,...), most of which are actually a lot more useful than merely keeping track of spammers and spambot infected hosts, and keep a type of offense (e.g. forum spammer, crawl rate abuse,) on top of IP addresses, hostnames, CIDR ranges,... in blacklists they publish as well. Some WAFs will come with the ability to connect to these databases, saving you the trouble of being targeted by the same actor that might have been already blacklisted for same detected activity on another web server.
Now, one thing needs to be said quite clearly - none of these methods can be considered bulletproof! They will remove the majority of offending web requests, which is valuable on its own and will let you focus better on those harder to detect offenders that somehow bypassed your protections.
There are of course countless techniques for both automated crawlers / content scrapers detection (and their own countermeasures - detection avoidance techniques) that I won't describe here, nor list all possible WAFs and their capabilities, not wanting test your patience or reach limits of the purpose of this Q&A. If you'd like to read more on what techniques can be employed to thwart against such unwanted visitors, then I recommend reading through the documentation on the OWASP Stinger and OWASP AppSensor projects.
Edit to add: Suggestions from HTTrack authors can be read in the HTTrack Website Copier FAQ: How to limit network abuse - Abuse FAQ for webmasters document, and the reasons why a single deterministic method of detection won't work (short of blacklisting offending IP addresses after the fact or through experience of other honeynets), if the adversary is set to obfuscate spider's
user agent string by setting it to any of the many user agent strings of real and legitimate web browsers, and disrespect
robots.txt directives, become rather apparent by glimpsing through the HTTrack Users Guide. To save you the bother of reading it, HTTrack includes simple configuration and command line flags to make it work in stealth mode and appear just as benign as any other legitimate user to simpler detection techniques.
1If a blacklisting was really so useful, Google would use it instead of simply limiting access to their google.maps. No matter how great you construct your blacklisting, you eventually end up aggravating and even alienating legitimate users. BTW: Making a buck was not Google main objective (not that they do not like money), but limiting abuse (automated data harvesting). And even use limits are easy to circumvent - different user IPs for once. So @CodesInChaos comment stands.– JeffzJul 6, 2013 at 15:26
@Jeffz - YMMV and is of course application specific. That said, I don't see the relevance of your comment. I (and many others) have already mentioned rate limiting or other time / client based quotas as a possible way to defend against content theft. And of course blacklist sensitivity CAN be dynamic, and entries can be automated, based on approaches I described. I disagree - blacklisting is useful, but of course to a limited extent. Please read at least the bolded parts of my answer, you might notice I've already mentioned it's hardly considered bulletproof. But it will help! ;) Jul 6, 2013 at 15:34
Everything the human user sees, he can record. As @Adnan points out, this is rather easy, and can be automated.
However, some sites still have some relative success at deterring mass slurping. Consider, for instance, Google Maps. Many people have, occasionally, tried to recover high-definition maps of large areas through scripting. Some have succeeded, but most were caught by Google's defences. It so happens that it is difficult to make an automatic downloader which acts, from the point of view of the server, as if it was under human control. Humans have all sorts of latencies and usage patterns which a shrewd sysadmin can notice and check on.
Similar tricks are done on, for instance, Stack Exchange. If you try to automate access to the site, you will soon be redirected to a page with a CAPTCHA.
Ultimately, this kind of security is not very satisfying because the defender and the attacker are on equal grounds: it is cunning against cunning. So, this is expensive: it requires thinking and maintenance. However, some sites do it nonetheless.
A generic way for attackers to defeat anti-automation safety measures is to "automate" the slurping with actual humans. Very cheap human workers can be hired in some countries.
1Or hire people to solve the CAPTCHAs– copyJul 5, 2013 at 14:13
5FWIW, we also blacklist offending IPs (at Stack Exchange).– SklivvzJul 5, 2013 at 14:18
1@Sklivvz What kind of moron tries to scrape SE if they could just download the dump made available anyway? Jul 6, 2013 at 14:08
2@TobiasKienzler the kind of moron that finds it easier to simply reskin a copy of the site instead of writing all the presentation layer... :-)– SklivvzJul 6, 2013 at 22:43
First of all, the only way you can prevent your site from being copied is actually never make it public to no one but you.
One way you could try to persuade people from doing it is with legal means, I'm not a lawyer so I don't know what steps you should take, if your content is original you could restrict the copyright or something similar.
I think that if you fear your site may get copied It has to be a really really really great web site.
Short answer, no, if the user loads a page, then the user can copy HTML by viewing the source.
If the website copier has a particular user agent, you can block that. See Stack Exchange for details.
Another solution might be to make a Flash webpage; those are hard to copy by hand anyways.
Or you can put some kind of human-verification on your site so a protected PHP directory include isn't called unless the user specifically clicks a non-link DOM object (like a line that says "enter here") that triggers the content to load.
First of all, as others have said - anything that you can see you can copy, using various methods. It depends why you want to prevent your website being copied, but the most effective method would probably be to add watermarks so that everyone knows where its come from. Perhaps even a polite notice asking people not to copy your website wouldn't go a miss.
However, going back to your original question and how to stop software from copying website, I believe CloudFlare has a web application firewall. I certainly know that Acunetix Web Vulnerability Scanner won't scan a website that uses CloudFlare. It's a free solution and it should also help speed your website up.
There is now foolproof solution though and anything can be circumvented. The best thing you can do is use a combination of the answers here, depending on how badly you need/want to protect your website. The best advice though, is if you don't want it copied, don't let people have it.
Disclaimer: this is an evil answer. I do not condone any of the following.
If you put all this together, and some more, you will find there are a number of layers of execution between the site's source code and the pixels that make up the letters and words that the user can read.
All these layers of execution can be obfuscated and/or exploited.
For example, you can generate markup that has little or no resemblance to the graphic output, to make looking at the HTML source of your website an exercise in futility. You could use one HTML tag per letter, reordering them with a creative use of
position:, hiding some of them with complex, generated CSS rules, and adding some more that weren't there, with CSS-generated content.
You can create a font that uses a custom mapping between character codes and glyphs, so that copy and pasting your content would yield utter garbage, or even swear words! You can split letters in two or more pieces and use Unicode combining characters to put them back together. You can do all this with a dynamic generator, creating a new random masterpiece of obfuscation for each HTTP request.
These are off the top of my head, there are (countably) infinite other ways to f*** with people's computers.
Now be a good boy/girl and take your blue pill :-)
You could take a look at David Madore's Book of Infinity, it's a small CGI program that generates an infinite number of pages to punish mass downloaders who don't respect robots.txt– lorebJul 6, 2013 at 12:06
@loreb - I honestly don't get it. So your website is being scraped by some cloud hosted crawler that disrespects your
robots.txtand you do a self-inflicting DDoS on your website as punishment for that crawler? How is that going to work? You realize that you'd only needlessly add to server load and exhaust its resources (CPU, memory, bandwidth,...), if the crawler is distributed, has seemingly limitless bandwidth and doesn't care about its crawl rate? You should drop its requests ASAP, not give it more work to do. Jul 6, 2013 at 14:21
@TidalWave sure, the book of infinity is a joke program, both in attitude (insistance on reproducing the same meaningless "book", and not just random content) and in practice, exactly as you described. That being said, if I were to take my suggestion seriously, I'd defend it stating that (1) the OP mentioned HTTrack in a way that suggests single users mass-downloading a website rather than a distributed crawler, and that (2) one could use the book of infinity to generate a tarpit, similar to OpenBSD's spamd.– lorebJul 7, 2013 at 15:34
Even AJAX with date parameters can be duplicated. I've scraped sites with heavy AJAX using GET/POST parameters. If I really need to emulate the browser I can just use selenium or something of that sort. I can always find a way to scrape a site if I really wanted to. Captcha is probably the most difficult thing to deal with. Even then there's the Captcha sniper and other modules to assist in these areas.
Look at this links you may get solution from this :)
Use robots.txt to prevent website from being ripped?
The simplest way is to identify the browser id who is browsing your page , if it is htttrack block it, ( you need to configure your server or use ur programming skill to load the different page accordingly )
3HTTrack is an open source application. You can easily modify the source and override any mechanism that respects
robots.txt.– AdiJul 7, 2013 at 15:40
There isn't a single deterministic method that would identify HTTrack clients if they're set to obfuscate their signature and disrespect
robots.txtdirectives. Not without resorting to a lot more advanced methods of detection. Quoting HTTrack user guide we get these two reasons why your suggestion wouldn't work: "The 'User Agent' field can be set to indicate whatever is desired to the server" for your suggestion on using UA, and "
robots.txtand meta robots tags (
2=always)" for blocking in
robots.txt. Jul 16, 2013 at 8:25