15

I was the target of different attacks today. The last one created this traffic on port 80. Apache is down and the load on server remains high.

Firewall is enabled. Any suggestions?

15:27:10.203993 IP 188.125.110.42.38818 > 190.10.34.115.http: Flags [S], seq 3395115767, win 29200, options [mss 1460,sackOK,TS val 22777069 ecr 0,nop,wscale 7], length 0
15:27:10.204050 IP 103.21.182.66.62502 > 190.10.34.115.http: Flags [S], seq 3158122291, win 8192, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
15:27:10.204079 IP 50.143.120.99.54622 > 190.10.34.115.http: Flags [S], seq 3624243404, win 29200, options [mss 1460,sackOK,TS val 273756706 ecr 0,nop,wscale 7], length 0
15:27:10.204128 IP 62.216.187.156.42248 > 190.10.34.115.http: Flags [S], seq 846253572, win 29200, options [mss 1460,sackOK,TS val 523062741 ecr 0,nop,wscale 7], length 0
15:27:10.204305 IP 23.105.195.199.38172 > 190.10.34.115.http: Flags [S], seq 2631287484, win 14600, options [mss 1460,sackOK,TS val 2125155826 ecr 0,nop,wscale 7], length 0
15:27:10.204774 IP 76.111.172.248.39602 > 190.10.34.115.http: Flags [S], seq 2199780458, win 29200, options [mss 1460,sackOK,TS val 249941894 ecr 0,nop,wscale 7], length 0
15:27:10.204797 IP 104.154.65.106.44962 > 190.10.34.115.http: Flags [S], seq 2397138535, win 28400, options [mss 1420,sackOK,TS val 1992688634 ecr 0,nop,wscale 7], length 0

Edit: This was our solution to the problem.

  • 1
    You tagged this as a DDoS. Are you asking how to stop a DDoS? – schroeder Jan 29 '17 at 15:15
  • 1
    IT's not a Synflood attack. It's this one .... hyperborea.org/journal/2014/03/pingback-ddos even with cloudflare activated still they reach the site and even we are blocking 3GB hour of pingback still we are working on it .... – antony Jan 29 '17 at 21:31
  • I have the full list of IPs that generate in our server many Gbyte of pinggback request. We had to block them to make the site up and running. How can I share the full list? – antony Jan 29 '17 at 22:48
  • 2
    If it's pingback attack, Cloudflare had a WAF ruleset to block pingback (blog.cloudflare.com/wordpress-pingback-attacks-and-our-waf) that you can enable if you get pingback attacks. – Lie Ryan Jan 30 '17 at 2:32
  • 3
    "Firewall is enabled." Can you give some details? Every firewall I know needs more work than just enabling it to work properly. – Num Lock Jan 30 '17 at 7:49
20

This is a classic SYN flood attack, preventive measures include

  • hashing syn packets, implementing syn cookies
  • checking if the client properly sends RST after an invalid packet sent back to him
  • ddos mitigation software often offer to time out specific syn packets based on advanced filtering
  • not storing the whole syn packet for further reference, only the IP and sequence number (also requires additional firewall/software to be installed on your part)
  • 7
    This is happening right now, you're talking about redesigning the server – schroeder Jan 29 '17 at 15:31
  • # /sbin/sysctl -w net.ipv4.tcp_syncookies=1 OR # echo 1 > /proc/sys/net/ipv4/tcp_syncookies CAN IT WORKS? Thank you! – antony Jan 29 '17 at 15:33
  • 1
    I know nothing about your system but I guess echo 1 > /proc/sys/net/ipv4/tcp_syncookies would work – Rápli András Jan 29 '17 at 15:37
  • 7
    It's very hard to defend against an ongoing attack, if you think your situation is critical, contact one of the ddos companies, implement Cloudflare, Insapsula, etc. – Rápli András Jan 29 '17 at 15:47
  • 2
    Technically speaking, it's still a synflood attack. It's over TCP and the attackers are initiating connections with SYN. – Rápli András Jan 29 '17 at 21:35
10

Limit NEW traffic

sudo iptables -A INPUT -p tcp --dport 80 -m state --state NEW -m limit --limit 50/minute --limit-burst 200 -j ACCEPT

Limit established traffic

sudo iptables -A INPUT -m state --state RELATED,ESTABLISHED -m limit --limit 50/second --limit-burst 50 -j ACCEPT

Enable syncookies on your kernel ( Recommanded by @Rápli András)

echo 1 > /proc/sys/net/ipv4/tcp_syncookies

Install and configure ddos-deflate

Notable Features:

  • It is possible to whitelist IP addresses, via /etc/ddos/ignore.ip.list.
  • It is possible to whitelist hostnames, via /etc/ddos/ignore.host.list.
  • Simple configuration file: /etc/ddos/ddos.conf
  • IP addresses are automatically unblocked after a preconfigured time limit (default: 600 seconds)
  • The script can run as a cron job at chosen frequency via the configuration file (default: 1 minute)
  • The script can run as a daemon at chosen frequency via the configuration file (default: 5 seconds)
  • You can receive email alerts when IP addresses are blocked.
  • Control blocking by connection state (see man netstat).
  • Auto-detection of firewall.
  • Support for APF, CSF and iptables.
  • Logs events to /var/log/ddos.log
  • Uses tcpkill to reduce the amount of processes opened by attackers.
5

SOLVED

In this case, it was a synflood attack with compromized Wordpress sites that generated tons of traffic (pingback) to our site at Gbytes/hour.

Here is the full description of the attack. In the referrer/header you can read "Wordpress" as the user agent: https://www.hyperborea.org/journal/2014/03/pingback-ddos/

Blocking these IPs with a firewall rule (see list in the link) solved the problem. I share the list, without any responsibility of curse.

LIST OF 5000 IPs involved to be blocked with firewall.

  • 5
    This doesn't provide any suggestions (as requested) for handling the situation, and as such is not an answer to the question as asked. See security.stackexchange.com/help/how-to-answer and security.stackexchange.com/help/self-answer in the help center. – a CVn Jan 30 '17 at 6:55
  • 1
    Ok maybe is missing the part where I should say, we Solved the problem blocking these IPS with a firewall rule. hope it helps. – antony Jan 30 '17 at 10:07
  • If we had this information and this list of IPs we would have solved the problem in less than 1 hour – antony Jan 30 '17 at 10:22
  • 1
    If this solved your problem, feel free mark it as accepted even though it is an answer to your own question (but you might have to wait a while). Or you can mark another answer as accepted, it is up to you. – Anders Jan 30 '17 at 12:09
2

The output you are showing suggests you are receiving a flood of SYN packets and not responding to any of them. Extrapolating from the numbers shown it amounts to about 7.5kpps. Even assuming a maximum packet size of 60 bytes of IPv4 header and 60 bytes of TCP header that still totals less than 8Mbit/s.

So if your network connection is 10Mbit/s or faster you should be able to sink that traffic assuming you have sufficient processing power to handle the number of packets per second.

From the limited information in your question it is impossible to tell what is causing the current load. It is a possibility that your firewall rules are structured in a way that cause each packet to consume a significant amount of CPU time in the firewall. If that's the case you need to restructure your firewall rules to be more efficient.

It seems you have configured a firewall to silently drop all of the packets. That can actually make the problem worse because legitimate clients will retransmit their SYN packets until you respond (or the client times out).

Rather than silently dropping the packets I recommend that you respond to as many of them as your upstream bandwidth permits. The responses would need to be either SYN-ACK packets or RST packets.

If you are experiencing a SYN flood where you cannot tell the difference between attack packets and legitimate packets, and if you need to keep serving requests from legitimate clients, I recommend that you enable SYN cookies as suggested in the answer by Rápli.

Under such circumstances SYN cookies can be the difference between your server responding slowly under the load or not responding at all.

Don't block IP addresses based on just the SYN packets. If you only see a SYN packet the source IP may be spoofed. And blocking based on that will do you no good and open you up to a different kind of DoS attack.

If a client completes a handshake and then sends bad traffic you can consider blocking the IP address. However doing so with IPv4 can lead to collateral damage, so it should be done only as a last measure. If you do block traffic that way make sure that you respond with RST packets, and make sure that your firewall rules are structured in a way that allow them to be evaluated with minimal CPU consumption.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.