4

Someone started attacking my site a few minutes ago and its caused PHP-FPM to max out all cores (4) on my vps and NGINX is now serving 502 to all users.

I'm seeing a bunch of these requests with tons of different agents

xx.xxx.xx.xx - - [10/Nov/2014:5:14:35 -0500] "POST / HTTP/1.1" 502 574 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/536.11 (KHTML, like Gecko) Chrome/20.0.1132.47 Safari/536.11"

I'm also seeing a bunch of requests from various WordPress sites

xxx.xx.xxx.xxx - - [10/Nov/2014:5:14:35 -0500] "GET / HTTP/1.0" 502 172 "-" "WordPress/3.9.2; http://www.xxxxxxx.com; verifying pingback from xx.xxx.xxx.xxx"

What kind of attack is this and how do I help mitigate it? Seems like some sort of botnet, its tons and tons of requests a second from all different IPs and agents.

I'm running a forum so my site heavily relies on nginx passing to php. I'm on NGINX 1.7.7 and PHP 5.6.2 if it matters. NGINX has practically zero load during the attack.

Edit: So apparently I didn't give enough information.

I run a pretty small forum - maybe 100-200 unique members a day. Normally very very low load - I have things cached and configured pretty well. Someone came on my site and TOLD me he was going to attempt to take down my website. A few minutes later, my logs were filled with hundreds (per second) of these POST and GETs from all different IPs and various wordpress blogs. The site started to serve 502 errors. I checked the server usage using HTOP and saw php5-fpm using almost 100% of all 4 cores of the VPS. I quickly shut down nginx and the load immediately went to zero. I left it off intentionally for a few hours and put it back up to find the attack had stopped.

While there is no active attack right now, i'd like to help mitigate what this user was able to do. I'm using OVHs network so I'm covered by their Anti DDoS VAC system - however this appears to be on the application level and obviously that wouldn't help me here.

I'm not too clever with web servers, so I don't know how to "check" what was in these post requests. All I see from my access logs are floods of unique IPs all doing POST to the root of my domain.

closed as unclear what you're asking by TildalWave, Xander, AJ Henderson, schroeder, Mark Nov 10 '14 at 20:42

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • Some additional questions I have / things to consider are: 1. Are you running a high profile website (e.g. like a bank or something like that)? There is a lot more motive to DDoS a high profile site, but that does not make lower profile sites immune. 2. Did you just put the site online? If the site has been up for a while, it is not as likely to be something wrong with the code, but it could be if it just went online. 3. Did you just update your code? If so, it is more likely that the update introduced a bug and that it is not a "real" DDoS attack. – Jonathan Nov 10 '14 at 16:52
  • 1
    "Wide range of user-agents" can be a sign of an accidental DDoS, such as what happens when your site is linked from a popular website such as Reddit. Are you logging referers, and if so, do they all come from the same place? – Mark Nov 10 '14 at 20:42
  • I have updated my question with more information and I hope it helps. – Maxriff Nov 11 '14 at 7:57
  • Oh, the joys of Wordpress. I assume you have total control of your VPS so that's a +1 for you, and I also assume you are not using any kind of "soft" firewall apart from the anti ddos service your provider offers to you, right? I would have lots to say on the matter, gonna expand it once the question gets unlocked again :) – MacK Nov 13 '14 at 8:30
3

DDoS attack related charactaristics:

This definitely has the characteristics of Distributed Denial of Service (DDoS) attack, and this is the first thing I suspect. The items that lead me to believe this are:

  1. Wide variety of sources (distributed)

I'm seeing a bunch of these requests with tons of different agents

(it appears in your log post the xxx.xxx.xxx.xxx would indicate that they are from a wide variety of IP addresses)

  1. Similarity of requests

I'm also seeing a bunch of requests from various WordPress sites

The similarity of the requests also leads me to believe Distributed Denial of Service. For example, the owner of a botnet could be issuing the command to attack your server with this specific type of request.

  1. "Flooding volume" of requests (denial of service)

Furthermore, the requests are of such volume that they are maxing out your resources. If this is a sharp increase of volume, it is likely to be an attack, or possibly a bug. It could also be that your site is becoming popular and you need more resources (especially if it has been a steady increase, and not a sharp spike in traffic).

How a DDoS attack can be performed: In a DDoS attack, oftentimes a botnet of innocent user's computers taken over by malware is employed by the malicious botnet user to attack a specific site. The flood of some type of request from many different sources is a characteristic of a DDoS attack.

Other possibilities:

Note: Your case appears to be a DDoS attack, and none of these other possibilities, based on your clarifications (especially since the guy said he was going to do it!)

  1. Bug in code: As TidalWave mentioned, it could possibly be something wrong in your code, but I tend to lean toward it really is a genuine DDoS attack, especially since it wasn't happening before (unless you just recently put your site online). One example I can think of where it could be your code is that your code sends something out to a bunch of sites that would make them send this traffic back to you. Perhaps you could be sending something out over broadcast when it shouldn't be broadcast. One thing to note is if this started happening right after you did an update, it would be more likely to be an issue in the code.

  2. Increase in website traffic: Your site may becoming more popular! This is more likely to be the case if your traffic has gone up steadily, and is finally maxing out your resources. However, a sharp spike in activity (not related to a specific "event" on your website, such as a contest drawing being held now) would more likely indicate a DDoS attack.

  3. It could be something else none of us have considered.

Mitigations:

Here are a few sites I found while researching preventing DDoS attacks: http://www.itbusinessedge.com/slideshows/show.aspx?c=96534 http://www.cisco.com/c/en/us/support/docs/security-vpn/kerberos/13634-newsflash.html http://www.networkworld.com/article/2170051/tech-primers/four-ways-to-defend-against-ddos-attacks.html

Here is what I gathered from them, as far as potential mitigations go:

  • Use an Internet Service Provider (ISP) that provides protection against DDoS, and that does not charge per bandwidth usage.
  • Adjust firewall settings (especially logging it for law enforcement)
  • Do it yourself (DIY) (probably not so good unless you are an expert in this area)
  • Enterprise DDoS prevention solutions (costly, and may still not work)
  • Cloud services designed to prevent DDoS attacks
  • FYI, I edited the answer to elaborate. It is possible it is not, but it does definitely have the characteristics of one. – Jonathan Nov 10 '14 at 16:45
  • 1
    I have updated my question with more information and I hope it helps. – Maxriff Nov 11 '14 at 7:57

Not the answer you're looking for? Browse other questions tagged or ask your own question.