7

An Ubuntu server of my company has been hacked to carry out a DoS attack. I found the shellshock bug had not been fixed by my colleagues, and I think it's the problem. Then, I found an ELF file that sends thousands messages, and the script is auto-generated by something. Even if I try to remove it, it creates newly by itself by using a new name (in /boot, /etc/init.d). Besides, I see the netstat command doesn't show me all real open ports. Maybe has the command been replaced? How is it possible to re-install it?

marked as duplicate by D.W., Mark, Gilles, TildalWave, Xander Nov 29 '14 at 16:01

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 54
    Wipe the machine and re-install it from scratch. – Stephen Touset Nov 28 '14 at 11:37
  • 1
    For now I cannot re-install the system since it's an important customer's server. We need to restore it, and soon we will schedule the configuration of a new server. – roghan Nov 28 '14 at 12:02
  • 23
    It's important to take the server offline as quickly as possible. You cannot trust any data passed through the server, and the attack may already have any data stored or accessible by the server. Further, it could be involved in an attack against another person (DDoS for example), and you wouldn't want to be participating in an attack. – Chris Murray Nov 28 '14 at 12:05
  • 12
    "For now I cannot re-install the system since it's an important customer's server" - you know how weird that sounds?. important customers' stuff should be on servers which can be rebuilt in seconds... see puppet/chef or any similar tool. whatever you do now, please wipe that machine later. – Karoly Horvath Nov 28 '14 at 18:09
  • 2
    Burnerate the server. – Zach Mertes Nov 28 '14 at 23:52
60

You should "nuke it from orbit": wipe and reinstall the OS and applications from clean source media, and then carefully restore the data from backup.

  • 6
    +1 This is the only really reliable way to deal with a compromised system. I would clarify that for anything possible, it would be better to reinstall software from scratch instead of using the back up. Only actual user-generated data should really need to come from the back up. – jpmc26 Nov 28 '14 at 18:53
8

It's always a question of effort vs result. If an attacker has gained complete control over the system, there are thousands of things that can be replaced or trojanized. Hunting them down one by one is a very time consuming task.

If the attackers aren't very sophisticated, you can start by searching for files that have been modified recently. One starting point is to look for files on your system that have last been modified in /usr/bin and /sbin (where most commands are).

ls -ltr /usr/bin

Check what's going on in /etc by looking at files that have been modified in the last 10 days:

find /etc -mtime -10

These are really basic first steps, and they don't cover looking at currently running (and potentially hidden) processes, that may hide any recent changes to the system. That's why this can turn into a cat and mouse game that can last for weeks if you miss just a single point of entry.

Unfortunately, reinstalling is the best option - least time consuming and almost guaranteed result!

  • 5
    that's provided ls and find have not been modified themselves. – njzk2 Nov 28 '14 at 15:30
  • Good point, that's why I mentioned in my original answer "if the attackers aren't very sophisticated"... :-) – Milen Nov 28 '14 at 15:32
  • 1
    otherwise you'll need to do the analysis from the outside, by mounting the disk on a clean machine (noexec obviously) and check it with clean tools. – njzk2 Nov 28 '14 at 15:36
  • 1
    What about a hard disk firmware hack? spritesmods.com/?art=hddhack&page=6 – ike Nov 28 '14 at 16:07
  • 4
    Assuming netstat and ps were compromised I'd say the attackers are a bit sophisticated (it's not your usual script kiddie IRC bot) and there is a good chance that find and ls are compromised as well. – user42178 Nov 28 '14 at 21:37

Not the answer you're looking for? Browse other questions tagged or ask your own question.