If I suspect my server is compromised then should I disconnect it immediately or not? From Power or Network?

Question

According to Mandiant, you should not:

Mistake #1: Immediately entering “remediaton mode” – also known as playing “Whack-A-Mole”
... as described here: http://blog.mandiant.com/archives/1525

Are there scenarios when it's prudent to take immediate action by disconnecting?

Merged with this question as the answers should be useful and similar:

When a server gets rooted (e.g. a situation like this), one of the first things that you may decide to do is containment. Some security specialists advise not to enter remediation immediately and to keep the server online until forensics are completed. Those advises are usually for APT. It's different if you have occasional Script kiddie breaches, so you may decide to remediate (fix things) early. One of the steps in remediation is containment of the server. Quoting from Robert Moir's Answer - "disconnect the victim from its muggers".

A server can be contained by pulling the network cable or the power cable.

Which method is better?

Taking into consideration the need for:

1. Protecting victims from further damage
2. Executing successful forensics
3. (Possibly) Protecting valuable data on the server

Edit: 5 assumptions

Assuming:

1. You detected early: 24 hours.
2. You want to recover early: 3 days of 1 systems admin on the job (forensics and recovery).
3. The server is not a Virtual Machine or a Container able to take a snapshot capturing the contents of the servers memory.
4. You decide not to attempt prosecuting.
5. You suspect that the attacker may be using some form of software (possibly sophisticated) and this software is still running on the server.

Answer 1

Similar to what @AviD said: if you can somehow determine that this compromised app/host/network is actively in progress with another attack, it might be wise to at least pull the network cable, right? This also assumes that you cannot control the attack in some other way, or that things have gotten extremely out of control.

This probably precludes any process or infrastructure, and also requires some very low skilled or unintellectual staff.

You know it's amateur hour when they pull the power, network, or disks on a compromised machine.

Answer 2

The answer depends on your level of sophistication, the level of sophistication of the attacker, and your goals.

The Mandiant blog post, from one of the leading providers of incident response and computer forensics services, is intended for sophisticated organizations responding to an Advanced Persistent Threat (APT). One of their concerns is that you might not actually even resolve the problem if you can't observe the attackers work across various systems on your network.

But most security incidents are less sophisticated attacks on poorly administered machines. In this case I think you're better off following the advice in the highly rated answer to the Server Fault question "My server's been hacked EMERGENCY", which suggests that disconnecting the server is indeed typically the first response, though you should not act in haste.

Answer 3

Useful answer over here by Robert Moir - some very good chat on Serverfault on this.

Answer 4

Yes, if you detect an active attack currently in progress, and your detection mechanism assures you that the attacker has not yet "pwned" your system, or achieved his objective.

Note that this might only be relevant if you have strong, very intelligent detection mechanisms in place, with inline correlation and active notifications.

Answer 5

RAM forensics (e.g. /dev/shm) can be helpful.

But I prefer unplugging the power cable (but try to log-in and rsync /proc right before).

The reasons for going for the power cable are:

1. When you do forensics in a hacked system, you are "stepping all over the crime scene"
2. The root kit keeps running - not so hard for the malicious to execute something (e.g. system wipe-out) on Network Link Down event.

Kyle Rankin gave a nice Intro to Forensics talk - there he recommends pulling the power cable.

Answer 6

It depends.

The Mandiant blog focuses on the APT. If that's your main concern, their advice may have merit.

However there are also many organizations where the APT is probably not the primary concern, and for those organizations, it may be reasonable to disconnect network access to a compromised machine as soon as you detect a compromise.

For instance, at one organization I know well, security is, quite reasonably, a relatively low priority. There's no way they're going to do a forensics investigation on each machine that's been compromised. They couldn't possibly afford it; and it would probably be too disruptive, too. If you know you're not going to do a forensic investigation on every compromised machine, there's little reason to delay pulling the plug. Similarly, they're not a likely target for APTs; for the most part, they don't have data of sufficient value. Their main challenge is routine everyday "script-kiddy" penetration of user-managed machines.

Answer 7

I totally agree with nealmcb.

1. If you are facing an "APT" and want to investigate on the compromised server, you can leave the server compromised and analyze the network traffic going from/to your server. This needs time to investigate and the hacker will continue to explore your IT infrastructure...

2. The common sense is to disconnect the server immediately to prevent unauthorized use of your infrastructure.

My advice would be to disconnect the server unless you have lot of time to fo forensics.