Is it possible to protect my site from HTTrack Website Copier or any similar program?
Without setting a max number of HTTP request from users.
No, there's no way to do it. Without setting connection parameter limits, there's even no way to make it relatively difficult. If a legitimate user can access your website, they can copy its contents, and if they can do it normally with a browser, then they can script it.
You might setup User-Agent restrictions, cookie validation, maximum connections, and many other techniques, but none will stop somebody determined to copy your website.
Protect the part of the site you want to protect with a username and password. Then only assign a username and password to people who sign an NDA, (or similar) that says they won't extract or copy information from your site.
Even then, nothing will prevent a determined skilled attacker from getting an offline copy, you can just make it harder so it's not worthwhile.
As has @Adnan already pointed out in his answer, there is really no way of stopping a determined person from copying snapshots of your website. I used the word snapshots here, because that's what such content scrapers (or harvesters) are really copying. They don't (or at least shouldn't) have access to your backend where your website contents are actually generated and displayed to the end user, so the best they can do is copy its output, one that you can generate in such a way to change in time or adjust according to its intended recipient (DRM schemes, watermarking,...), as has @makerofthings7 pointed out in his answer.
So this much about what's already been answered. But there is one thing about this threat that I feel haven't yet been well covered in mentioned answer. Namely, most of such content scraping is done by opportunistic and automated web crawlers, and we see targeted attacks a lot rarer. Well, at least in numbers - bear with me.
These automated crawlers can actually be blacklisted quite effectively through the use of various WAFs (some might even use honeypots to determine the threats in heuristic ways) that keep updated database of blacklisted domains (CBLs or Community Ban Lists, DBLs or Domain Block Lists, DNSBLs or DNS-based Blackhole Lists,...) where such automated content scrapers are operating from. These WAFs will deny or grant access to your content serving web application based on three main approaches:
Deterministic blacklisting: These are detections based on characteristics of web requests that content scrapers will make. Some of them are: Request originating IP address, Reverse DNS resolved remote hostname, Forward-confirmed reverse DNS lookup (see explanation in one of my questions here), User agent string, Request URL (your web application could for example hide a honeytrap URL address that a content scraper might follow in one of its responses, after it determines the request didn't come from a whitelisted address such as legitimate search engine crawlers / spiders)... and other fingerprint information associated with automated web requests.
Heuristic blacklisting: This is a way to determine a threat either by weighting parameters of a single web request described in the deterministic approach (anti-spam filters use a similar approach based on calculating Bayesian probability), or by analyzing multiple web requests, such as: Request rate, Request order, Number of illegal requests,... that might help determine, if the request comes from a real and intended user, or some automated crawler.
External DNSBL/CBL/DBLs: I've already mentioned relying on external DNSBL/CBL/DBLs (e.g. Project Honey Pot, Spamhaus, UCEPROTECT,...), most of which are actually a lot more useful than merely keeping track of spammers and spambot infected hosts, and keep a type of offense (e.g. forum spammer, crawl rate abuse,) on top of IP addresses, hostnames, CIDR ranges,... in blacklists they publish as well. Some WAFs will come with the ability to connect to these databases, saving you the trouble of being targeted by the same actor that might have been already blacklisted for same detected activity on another web server.
Now, one thing needs to be said quite clearly - none of these methods can be considered bulletproof! They will remove the majority of offending web requests, which is valuable on its own and will let you focus better on those harder to detect offenders that somehow bypassed your protections.
There are of course countless techniques for both automated crawlers / content scrapers detection (and their own countermeasures - detection avoidance techniques) that I won't describe here, nor list all possible WAFs and their capabilities, not wanting test your patience or reach limits of the purpose of this Q&A. If you'd like to read more on what techniques can be employed to thwart against such unwanted visitors, then I recommend reading through the documentation on the OWASP Stinger and OWASP AppSensor projects.
Edit to add: Suggestions from HTTrack authors can be read in the HTTrack Website Copier FAQ: How to limit network abuse - Abuse FAQ for webmasters document, and the reasons why a single deterministic method of detection won't work (short of blacklisting offending IP addresses after the fact or through experience of other honeynets), if the adversary is set to obfuscate spider's
user agent string by setting it to any of the many user agent strings of real and legitimate web browsers, and disrespect
robots.txt directives, become rather apparent by glimpsing through the HTTrack Users Guide. To save you the bother of reading it, HTTrack includes simple configuration and command line flags to make it work in stealth mode and appear just as benign as any other legitimate user to simpler detection techniques.
Everything the human user sees, he can record. As @Adnan points out, this is rather easy, and can be automated.
However, some sites still have some relative success at deterring mass slurping. Consider, for instance, Google Maps. Many people have, occasionally, tried to recover high-definition maps of large areas through scripting. Some have succeeded, but most were caught by Google's defences. It so happens that it is difficult to make an automatic downloader which acts, from the point of view of the server, as if it was under human control. Humans have all sorts of latencies and usage patterns which a shrewd sysadmin can notice and check on.
Ultimately, this kind of security is not very satisfying because the defender and the attacker are on equal grounds: it is cunning against cunning. So, this is expensive: it requires thinking and maintenance. However, some sites do it nonetheless.
A generic way for attackers to defeat anti-automation safety measures is to "automate" the slurping with actual humans. Very cheap human workers can be hired in some countries.
First of all, the only way you can prevent your site from being copied is actually never make it public to no one but you.
One way you could try to persuade people from doing it is with legal means, I'm not a lawyer so I don't know what steps you should take, if your content is original you could restrict the copyright or something similar.
I think that if you fear your site may get copied It has to be a really really really great web site.
Short answer, no, if the user loads a page, then the user can copy HTML by viewing the source.
If the website copier has a particular user agent, you can block that. See Stack Exchange for details.
Another solution might be to make a Flash webpage; those are hard to copy by hand anyways.
Or you can put some kind of human-verification on your site so a protected PHP directory include isn't called unless the user specifically clicks a non-link DOM object (like a line that says "enter here") that triggers the content to load.
First of all, as others have said - anything that you can see you can copy, using various methods. It depends why you want to prevent your website being copied, but the most effective method would probably be to add watermarks so that everyone knows where its come from. Perhaps even a polite notice asking people not to copy your website wouldn't go a miss.
However, going back to your original question and how to stop software from copying website, I believe CloudFlare has a web application firewall. I certainly know that Acunetix Web Vulnerability Scanner won't scan a website that uses CloudFlare. It's a free solution and it should also help speed your website up.
There is now foolproof solution though and anything can be circumvented. The best thing you can do is use a combination of the answers here, depending on how badly you need/want to protect your website. The best advice though, is if you don't want it copied, don't let people have it.
Disclaimer: this is an evil answer. I do not condone any of the following.
If you put all this together, and some more, you will find there are a number of layers of execution between the site's source code and the pixels that make up the letters and words that the user can read.
All these layers of execution can be obfuscated and/or exploited.
For example, you can generate markup that has little or no resemblance to the graphic output, to make looking at the HTML source of your website an exercise in futility. You could use one HTML tag per letter, reordering them with a creative use of
position:, hiding some of them with complex, generated CSS rules, and adding some more that weren't there, with CSS-generated content.
You can create a font that uses a custom mapping between character codes and glyphs, so that copy and pasting your content would yield utter garbage, or even swear words! You can split letters in two or more pieces and use Unicode combining characters to put them back together. You can do all this with a dynamic generator, creating a new random masterpiece of obfuscation for each HTTP request.
These are off the top of my head, there are (countably) infinite other ways to f*** with people's computers.
Now be a good boy/girl and take your blue pill :-)
Even AJAX with date parameters can be duplicated. I've scraped sites with heavy AJAX using GET/POST parameters. If I really need to emulate the browser I can just use selenium or something of that sort. I can always find a way to scrape a site if I really wanted to. Captcha is probably the most difficult thing to deal with. Even then there's the Captcha sniper and other modules to assist in these areas.
Look at this links you may get solution from this :)
The simplest way is to identify the browser id who is browsing your page , if it is htttrack block it, ( you need to configure your server or use ur programming skill to load the different page accordingly )
protected by Community♦ Jun 1 '14 at 15:52
Thank you for your interest in this question.
Because it has attracted low-quality or spam answers that had to be removed, posting an answer now requires 10 reputation on this site (the association bonus does not count).
Would you like to answer one of these unanswered questions instead?