4

I am getting slammed hard by bots, and I am not sure how to stop them without blocking my legitimate traffic. I created some iptables rules that say, if X connections in X seconds, block, but that doesn't do me much good when the bot is only making 2 or 3 requests every 30 seconds.

Ideally, I would like to identify these bots by the URLs and how many times they are hitting that URL, I am sure I can do this with some code, but I was wondering if anyone out there already has a solution that doesn't involve code, and preferably blocks via iptables.

Any ideas, thoughts, suggestions are greatly appreciated.

Here is the output from my access log:

27.153.233.93 - - [02/Nov/2013:10:44:54 +0000] "GET /register/ HTTP/1.1" 200 8296
192.99.2.188 - - [02/Nov/2013:10:44:54 +0000] "GET /register/ HTTP/1.1" 200 8296
110.86.164.186 - - [02/Nov/2013:10:44:54 +0000] "GET /register/ HTTP/1.1" 200 8296
121.205.239.188 - - [02/Nov/2013:10:44:54 +0000] "POST /register/ HTTP/1.1" 44 339
27.153.233.93 - - [02/Nov/2013:10:44:54 +0000] "POST /register/ HTTP/1.1" 44 339
36.251.24.146 - - [02/Nov/2013:10:44:54 +0000] "GET /wp-login.php HTTP/1.1" 200 1655
110.86.164.186 - - [02/Nov/2013:10:44:55 +0000] "POST /register/ HTTP/1.1" 44 339
192.99.2.188 - - [02/Nov/2013:10:44:54 +0000] "POST /register/ HTTP/1.1" 302 20
110.85.107.18 - - [02/Nov/2013:10:44:56 +0000] "GET /register/ HTTP/1.1" 200 8296
110.86.187.169 - - [02/Nov/2013:10:44:56 +0000] "GET /register/ HTTP/1.1" 200 8296
125.117.214.174 - - [02/Nov/2013:10:44:55 +0000] "POST /wp-admin/admin-ajax.php HTTP/1.1" 200 1166
110.85.107.18 - - [02/Nov/2013:10:44:56 +0000] "POST /register/ HTTP/1.1" 44 339
110.86.187.169 - - [02/Nov/2013:10:44:56 +0000] "POST /register/ HTTP/1.1" 44 339
192.99.2.188 - - [02/Nov/2013:10:44:55 +0000] "GET / HTTP/1.1" 200 10718
110.89.13.58 - - [02/Nov/2013:10:44:57 +0000] "GET /register/ HTTP/1.1" 200 8296
36.251.24.146 - - [02/Nov/2013:10:44:57 +0000] "POST /wp-login.php HTTP/1.1" 302 20
110.89.13.58 - - [02/Nov/2013:10:44:57 +0000] "POST /register/ HTTP/1.1" 44 339
218.86.50.83 - - [02/Nov/2013:10:44:58 +0000] "GET /register/ HTTP/1.1" 200 8296
218.86.50.83 - - [02/Nov/2013:10:44:58 +0000] "POST /register/ HTTP/1.1" 44 339
117.26.195.51 - - [02/Nov/2013:10:44:58 +0000] "GET /register/ HTTP/1.1" 200 8296
120.43.20.103 - - [02/Nov/2013:10:44:58 +0000] "GET /register/ HTTP/1.1" 200 8296
121.205.196.193 - - [02/Nov/2013:10:44:58 +0000] "GET /register/ HTTP/1.1" 200 8296
117.26.195.51 - - [02/Nov/2013:10:44:58 +0000] "POST /register/ HTTP/1.1" 44 339
120.43.20.103 - - [02/Nov/2013:10:44:58 +0000] "POST /register/ HTTP/1.1" 44 339
120.37.207.234 - - [02/Nov/2013:10:44:59 +0000] "GET /register/ HTTP/1.1" 200 8296
27.153.210.2 - - [02/Nov/2013:10:44:59 +0000] "GET /register/ HTTP/1.1" 200 8296
121.205.196.193 - - [02/Nov/2013:10:44:59 +0000] "POST /register/ HTTP/1.1" 44 339
120.37.207.234 - - [02/Nov/2013:10:44:59 +0000] "POST /register/ HTTP/1.1" 44 339
120.37.206.69 - - [02/Nov/2013:10:44:59 +0000] "GET /register/ HTTP/1.1" 200 8296
120.37.206.69 - - [02/Nov/2013:10:44:59 +0000] "POST /register/ HTTP/1.1" 44 339
27.159.234.10 - - [02/Nov/2013:10:44:59 +0000] "GET /register/ HTTP/1.1" 200 8296
27.159.234.10 - - [02/Nov/2013:10:45:00 +0000] "POST /register/ HTTP/1.1" 44 339
Improve this question
7

It appears that Fail2ban fits your scenario perfectly. It is rather configurable so you should be able to setup what you need by consulting the documentation a little.

Improve this answer
  • fail2ban is good for this, though you have to tune the number of attempts pretty high to avoid blocking legitimate people who can't read your CAPTCHA or have other trouble registering. – Michael Hampton Nov 2 '13 at 18:41
4

You don't have much information in that log, so you need to assume something (or use some tool that will do that for you).

For example, say that most of your clients are in US/EMEA. Then maybe all IPs coming from China can be blocked. You can get them with WHOIS from the logs, or try online services:

Manually (with the advantage of only blocking really "guilty" subnets):

$ whois 218.86.50.83 | grep "country\|inetnum" | sort | uniq
country:        CN
inetnum:        218.85.0.0 - 218.86.127.255
# iptables-deny 218.85.0.0/15

$ # Repeat for all "suspicious" IPs in your access.log

To find out who's hitting you, you can do a statistic on the access.log and the first bits of the incoming IP, grouping by the hour:

$ grep "GET /register/" /path/to/access.log| cut -f 1,4 -d " " | cut -f1-2 -d: | sed -e 's/\(.*\)\.[0-9]* .*:\(.*\)/\2 \1/g' | sort | uniq -c | sort -n

If you wanted to group by broader networks, 

$ grep "GET /register/" /path/to/access.log | cut -f 1,4 -d " " | cut -f1-2 -d: | sed -e 's/\(.*\)\.[0-9]*\.[0-9]* .*:\(.*\)/\2 \1/g' | sort | uniq -c | sort -n

This for your sample gives you

  1 10 110.85
  1 10 110.89
  1 10 117.26
  1 10 120.43
  1 10 121.205
  1 10 192.99
  1 10 218.86
  1 10 27.159
  2 10 110.86
  2 10 120.37
  2 10 27.153

and you can now look for, say, ^27.153 in the access.log (still China, still covered by ip2location).

Improve this answer
  • Thank you for your answer, Fail2ban worked out, but I also blocked all of china =] – Jeffrey L. Roberts Dec 7 '13 at 3:44
3

It would be much better to stop such kind of attacks at the perimeter rather than on the server itself. Most of the application layer firewalls come with IP reputation list. I don't want to recommend any specific product but CISCO, Juniper and most of the other vendors providing the firewall and IDP functionality provide IP reputation services as well. Consult your network team and they can provide you the specific settings you need.

Also, you can use Mod Security which is an open source application layer firewall. The advantage of mod security is that apart from the wide range of attacks it can stop, it also has the ability to log the complete HTTP POST requests. Apache don't do this by default. Once you have the complete packet, you can analyse it for any attack signature as well. For example, if the bot happen to use a particular POST parameter or packet header length etc, you can develop attack signatures based upon those parameters as well.

Improve this answer
2

if those URLS are legit, fail2ban like terry mentioned would be the easiest way to go. another way, but much more complicated in terms of setup and maintainance would be to use snort/suricata; it has thresholds/rate-limiting too, and ip-based blocks might be canceled after a certain amount of time.

problem occurs when you have just 1 or 2 hits per bot on those urls; the only chance to block malicious attempts without affecting regular users i can think of (but might affect legit users)

  • change /register/ to /register-humans-only/ in you application
  • block anybody who tries to use /register/

btw, your access-log seesm to miss some values, like referer and user-agent. can you see a pattern, based on the user-agent, e.g., is it always the same? if so, you could also try to block based on uri/user-agents.

if those URLs are not legit, just block anybody who wants to acces those.

Improve this answer
1

As mentioned, you can block entire country and you can also block IPs from servers: From the Amazon Datacenter as example.

I am using the list of IPs provided by : http://www.wizcrafts.net/

You can use a fail2ban filter like this one :

[Definition]
failregex = ^ .* POST /register.*
            ^ .* PUT /register.*

and the corresponding configuration file: 

[register]
enabled  = true
port     = http,https
filter   = wp-login
action = iptables-multiport[name=wp-login, port="http,https", protocol=tcp]
logpath  = /var/log/.../*access.log
bantime = 36000
findtime = 172800
maxretry = 4
Improve this answer

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.