An Ubuntu server of my company has been hacked to carry out a DoS attack. I found the shellshock bug had not been fixed by my colleagues, and I think it's the problem. Then, I found an ELF file that sends thousands messages, and the script is auto-generated by something. Even if I try to remove it, it creates newly by itself by using a new name (in /boot, /etc/init.d). Besides, I see the netstat command doesn't show me all real open ports. Maybe has the command been replaced? How is it possible to re-install it?
-
54Wipe the machine and re-install it from scratch.– Stephen TousetNov 28, 2014 at 11:37
-
1For now I cannot re-install the system since it's an important customer's server. We need to restore it, and soon we will schedule the configuration of a new server.– roghanNov 28, 2014 at 12:02
-
23It's important to take the server offline as quickly as possible. You cannot trust any data passed through the server, and the attack may already have any data stored or accessible by the server. Further, it could be involved in an attack against another person (DDoS for example), and you wouldn't want to be participating in an attack.– Chris MurrayNov 28, 2014 at 12:05
-
12"For now I cannot re-install the system since it's an important customer's server" - you know how weird that sounds?. important customers' stuff should be on servers which can be rebuilt in seconds... see puppet/chef or any similar tool. whatever you do now, please wipe that machine later.– Karoly HorvathNov 28, 2014 at 18:09
-
2Burnerate the server.– Zoey MertesNov 28, 2014 at 23:52
|
Show 6 more comments
2 Answers
You should "nuke it from orbit": wipe and reinstall the OS and applications from clean source media, and then carefully restore the data from backup.
-
6+1 This is the only really reliable way to deal with a compromised system. I would clarify that for anything possible, it would be better to reinstall software from scratch instead of using the back up. Only actual user-generated data should really need to come from the back up.– jpmc26Nov 28, 2014 at 18:53
It's always a question of effort vs result. If an attacker has gained complete control over the system, there are thousands of things that can be replaced or trojanized. Hunting them down one by one is a very time consuming task.
If the attackers aren't very sophisticated, you can start by searching for files that have been modified recently. One starting point is to look for files on your system that have last been modified in /usr/bin and /sbin (where most commands are).
ls -ltr /usr/bin
Check what's going on in /etc by looking at files that have been modified in the last 10 days:
find /etc -mtime -10
These are really basic first steps, and they don't cover looking at currently running (and potentially hidden) processes, that may hide any recent changes to the system. That's why this can turn into a cat and mouse game that can last for weeks if you miss just a single point of entry.
Unfortunately, reinstalling is the best option - least time consuming and almost guaranteed result!
-
5
-
Good point, that's why I mentioned in my original answer "if the attackers aren't very sophisticated"... :-)– MilenNov 28, 2014 at 15:32
-
1otherwise you'll need to do the analysis from the outside, by mounting the disk on a clean machine (noexec obviously) and check it with clean tools.– njzk2Nov 28, 2014 at 15:36
-
1What about a hard disk firmware hack? spritesmods.com/?art=hddhack&page=6– ikeNov 28, 2014 at 16:07
-
4Assuming netstat and ps were compromised I'd say the attackers are a bit sophisticated (it's not your usual script kiddie IRC bot) and there is a good chance that find and ls are compromised as well.– user42178Nov 28, 2014 at 21:37