I keep seeing "python-requests/1.1.0 CPython/2.7.4 Linux/3.8.0-19-generic" requests e.g. from Amazon's servers of my website. It's kind of creepy. Who would set up an Amazon server to scan websites? Should I just block them? Cheers
You have a Web site; it is meant to provide pages to whoever asks for them. That's the whole point of a Web site. What sense would it make to refuse to send the page to some people ? Especially if the exclusion criterion is the
User-Agent string, which is freely chosen by the client. Any individual with nefarious intentions can masquerade his software so that it would appear, on your side, as a basic Internet Explorer or Chrome or whatever; even if it was sensible to exclude HTTP clients which happen to be written in Python, filtering on the User-Agent would simply reject clients who are honest enough to admit using Python as their programming language.
It is plausible that this specific client is some sort of Web crawler. You cannot really prevent Web crawlers from slurping the data from your Web site, as long as you allow browsing from unauthenticated users, though you can install some mitigation measures against systematic site-wide download (that's what Google does, for Google Maps: if they see too many requests from a given client within a given time frame, they suspect foul play). You can document your unwillingness to see your site inspected by Web crawlers, with a
robots.txt file; most crawlers honour that file. But this won't stop any determined crawler. Best course of action is to:
- either implement and deploy authentication procedures, refusing access to people who have not been duly authenticated with, e.g., a user name or password;
- or accept that public data is public, and there is no such thing as "public except for evil people".