I want to prevent robots flooding my website and I'm thinking about a ban in these cases...

Can I assume that an empty user agent will always be a robot?

  • Can you describe what exactly you consider a robot? If you want to ban anything but common browsers then blocking an empty user-agent might help you a bit. But this will only help against a few non-browsers, most HTTP clients actually set some User-Agent and often they set it to something which looks like a browser in order to bypass simple restrictions. – Steffen Ullrich Oct 4 '18 at 21:47
  • 1
    I consider a "not human" :) – Karl Zillner Oct 4 '18 at 22:32
  • The User-Agent header, like any other HTTP header is under complete control of the client, so can be spoofed in many ways. Proxies may also "sanitize" it. So your assumption does not work and even if it does many other robots will come to your site with a non empty User-Agent header. – Patrick Mevzek Oct 4 '18 at 22:35
  • I know. But I was thinking....if someone make a fast try....like a curl in a terminal....and receive a block message. This guy doesnt know why Im blocking him. And it will be the first alert :) like "I know you are not doing something good here and Im watching you" – Karl Zillner Oct 4 '18 at 22:39
  • If an empty UA is always some code and never a human, I can ban him for some hours extending the second attempt to find out why he was banned – Karl Zillner Oct 4 '18 at 22:42

Almost always, but not always. It could be a user behind a misconfigured proxy, or a browser extension, or (more likely) someone using telnet/netcat:

netcat server 80
GET / HTTP/1.0

And done.

You can ban users without User-Agent set, but it's dead easy to set one. curl, wget, aria2, python and almost every single command/library/framework out there will have an option to override the User-Agent string.

You'd better implement rate-limiting on sensible pages, captchas when the limit is reached, and ban after a number of captcha misses.

| improve this answer | |
  • Yes...Its not the only rule, but I think if first access from a user is without user agent....hes just trying to use some curl thing. – Karl Zillner Oct 4 '18 at 22:29
  • curl have a defined user-agent string. It's something like User-Agent: curl/7.12.3 – ThoriumBR Oct 4 '18 at 22:33
  • I've already set a rate limit. I'm just trying to improve my security – Karl Zillner Oct 4 '18 at 22:34
  • Yes....I search for some words in UA and ban them....like sqlmap, curl, downloader, trojan and anothers.... – Karl Zillner Oct 4 '18 at 22:36
  • I voted this answer up, but I think that the “almost always” statement at the beginning should be removed. If a bot tries to access a website for some reason, its creator most likely specified a user-agent for it. Furthermore most programs you mentioned have default user agents, like the request lib in python. – Pedro Gomes Oct 5 '18 at 5:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.