0

I keep seeing "python-requests/1.1.0 CPython/2.7.4 Linux/3.8.0-19-generic" requests e.g. from Amazon's servers of my website. It's kind of creepy. Who would set up an Amazon server to scan websites? Should I just block them? Cheers

Improve this question
3
  • 2
    Can be anyone, you can block them but they'll just change their useragent to IE, or Firefox. I guess it's some kind of Ad service, maybe website indexing or archiving.
    – Stolas
    Commented Aug 22, 2013 at 6:10
  • 3
    It's not creepy unless you have ophidiophobia. User-Agent string isn't correlated with bad guys.
    – Deer Hunter
    Commented Aug 22, 2013 at 8:37
  • I HAVE set up Amazon servers to scan websites using Python ... Researchers, spammers, and botnets use Amazon as a cheap and easy way to maintain a server.
    – schroeder ♦
    Commented Aug 22, 2013 at 15:05

1 Answer 1

Reset to default
5

You have a Web site; it is meant to provide pages to whoever asks for them. That's the whole point of a Web site. What sense would it make to refuse to send the page to some people ? Especially if the exclusion criterion is the User-Agent string, which is freely chosen by the client. Any individual with nefarious intentions can masquerade his software so that it would appear, on your side, as a basic Internet Explorer or Chrome or whatever; even if it was sensible to exclude HTTP clients which happen to be written in Python, filtering on the User-Agent would simply reject clients who are honest enough to admit using Python as their programming language.

It is plausible that this specific client is some sort of Web crawler. You cannot really prevent Web crawlers from slurping the data from your Web site, as long as you allow browsing from unauthenticated users, though you can install some mitigation measures against systematic site-wide download (that's what Google does, for Google Maps: if they see too many requests from a given client within a given time frame, they suspect foul play). You can document your unwillingness to see your site inspected by Web crawlers, with a robots.txt file; most crawlers honour that file. But this won't stop any determined crawler. Best course of action is to:

  • either implement and deploy authentication procedures, refusing access to people who have not been duly authenticated with, e.g., a user name or password;
  • or accept that public data is public, and there is no such thing as "public except for evil people".
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .