Protecting against DOS attacks

Question

Our current site is easily DOS'd. I can perform an AB test against it from Apache and it disappears for about 5-10 minutes. We have been having problems with others DOSing the site - to the point where we had to shut it down for 8 hours (recommended by the server host).

The server is hosted by Rackspace. They recommended shutting it down for 8 hours, which when we brought it back up the attacked resumed.

We setup cloud flare for the DNS, however after an AB test with 58,000 requests sent the site was offline again (not as long as originally).

Rackspace is not offering anymore suggestions, and I need to figure out something to do for the time being, switching server hosts is not an option at this point and time (lead developer is leaving in 1 week for a new position).

This specific attack created a lot of connections in the mysql database with the command sleep.

Does anyone have any suggestions or recommendations on ways to protect again DOS attacks?

Would increasing the RAM on the virtual host help at all?

Answer 1

If the attack is creating a lot of database connections with a sleep command, it sounds like the DoS is not your direct problem, but rather a consequence of the problem. Which is, namely, that your database is sleeping a lot.

I would say there are two possiblities here:

• First possibility is that this is a built-in function, wherein certain circumstances, your application is explicitly causing the database to sleep (as @Rory mentioned). If this is the case, this is a very bad pattern, and should be removed. There are other solutions, and you should definitely avoid this.
• More likely, and even worse, is that you have a SQL Injection vulnerability in your website. The good news is that so far, it seems the attackers are only using it to DoS your site. That bad news is that it is probably much worse than that.
  But, more good news, it is a specific bug in your code that should be easily found and fixed by any competent programmer, and not an amorphous and anonymous "DoS" (which could really be any number of things).

Remediation?
First off, figure out which of the above is the case. Should be easy enough, start by asking the programmer if he ever sends a sleep command to the db on purpose. Also should be easy enough to search the codebase for relevant sleep strings.
If it's the first scenario - simply refactor the code to remove the offending command.
Otherwise - and you should probably do this anyway - read up a bit on SQL Injection, and get all the programmers to do likewise. Find and fix the flaws, then do a focused penetration test. (You can get an experienced security pro to help here). Then consider the root cause of such critical security bugs, and start enhancing the overall level of security, via training, code reviews, SDL, pentests, etc. (A security pro can help with this too).

Answer 2

There are a couple of things you can do to prevent DOS/DDOS.
First I would recommend using an autoban firewall. I've been using Fail2Ban for some time now. Most of my problems with DDOS/DOS attacks to SSH,FTP,BIND and etc were solved. What fail2ban actually does is it's scanning a log file and when a regex pattern matches X times it bans the person. With a little more work you can make it read almost any type of log files.

Second there were a couple of mods for apache that prevent DDOS/DOS attacks. One of them was mod_evasive. I used it for a while but fail2ban worked for me best (for now) so I didn't quite get into mod_evasive.

Another possibility is to use Nginx as a front-end which would proxy-forward connection requests to your Apache server. Then set Nginx's HttpLimitZoneModule (info) to a desired number. This will reduce the maximum incoming connection from a single IP but it won't prevent DDOS.

Finally - close all of your unused ports. Leave only ports that need to be public as port 21,53,80,443 etc. This will limit the possibilities where the attacker may target.

Edit:
Make sure you don't get ping flood by setting some iptables rules that limit icmp. Here's a link which explains how ping flood works. Try looking at the pictures first then the text will make more sense to you.
If you're using iptables this command may help you prevent ping/icmp flood:

iptables -A INPUT -p icmp -m limit --limit 6/s --limit-burst 1 -j ACCEPT
iptables -A INPUT -p icmp -j DROP

The first one will limit icmp requests up to 6 requests per second. Afterwords the person will not receive any more responses. This will eliminate bots that are doing a cycle:

for(i=0; i<1000000; i++){ 
   some evil code that will bring your server down 
   either by pinging or just connecting to random ports. 
};

Answer 3

This article is a good start for the layman. It ultimately suggests that you hire a security expert.

If that's not an option (if you are expected to be the expert), then I have read that an effective DOS mitigation strategy involves:

• Having cutting-edge IDS tools (3rd party or even homegrown) to detect the onset of a DOS attack (a sudden spike in traffic, etc.); and
• Devoting someone full-time (at least, for a while) to monitoring network traffic and blocking IP addresses that are connected to the server redundantly (more than once) --> see this article for more info

Without knowing who you are/work for, it is hard to guess whether you are being consistently re-attacked by the same parties. If you're a big firm, its probably multiple (disparate) groups trying to shut you down. If you're smaller, it's probably the same entity/group attacking you. If that's the case, then if you commit to aggressively blocking violating IPs in this fashion for several weeks, chances are you will grind the attackers down over time and thwart them.

The big thing in security is to make the cost of acquiring the asset greater than the value of the asset itself.

And, if you're a big company, you can afford to hire a true expert/consultant who knows leagues more than I do ;-)

Answer 4

Have a read of this question for some more information on how DoS attacks work.

For this specific case, the attack is using a function you are allowing (sleep) to force wait cycles until your database can no longer respond.

A solution for you may be to disallow the sleep command entirely, unless you have a need for it. A better solution is to prevent all connection to your database except through stored procedures, so that only those commands you need to run your application are allowed. This will make it that much harder for an attacker to find and exploit a hole.