How do you explain the necessity of "nuke it from orbit" to management and users?

Question

When a machine has been infected with malware, most of us here immediately identify the appropriate action as "nuke it from orbit" - i.e. wipe the system and start over. Unfortunately, this is often costly for a company, especially if backups are configured in a less-than-optimal fashion. This can produce resistance from management and users who just want to carry on using the machine for work. After all, as far as they're concerned, they can "just run AV over it" and everything will be fine.

How do you explain the problem to management and users of a non-technical nature? I could easily produce a technical reason, but I'm having trouble coming up with the appropriate wording for non-technical users. I'd especially appreciate any way of speaking that the recipient can identify with, e.g. appealing to a manager's sense of risk management.

Answer 1

In my experience management doesn't like to listen to clever analogies. Depending on the person they care about the bottom line in dollars or hours of productivity. I would explain:

The actual bottom line is that a compromise of our data will cost the company approximately X dollars + Y hours to recover. This is Z% likely to happen given the malware that is on this machine. A new install will cost A dollars + B hours to recover. You pick the appropriate action.

It's short and clear and doesn't really leave them any room to argue. They will clearly understand the risk and should make the right decision.

Answer 2

I would avoid the biological or non-business analogies (unless this is a hospital). Your job is to assess risk, cost, and provide options. Your management's job is to make the decision based on your analysis and advice.

Generally, an approach in a tabular format is best. "approach", "likelihood of correcting the problem", "cost" are the minimum needed. You can call the second "Vegas" if you absolutely have to get cute.

For example, in this case, you may have the following.

Approach	Prognosis	Cost
Run anti-virus on machine	30%	4 hours IT, 4 hours downtime
Replace machine with new machine	75%	$3,000, 16 hours IT, 4 hours downtime
AV machine, copy user files, replace machine, restore files	60%	$3,000, 24 hours IT, 4 hours user, 8 hours downtime

In this list (assuming a user desktop), the real problem is user behavior. You'll want to document why the prognosis is < 100% for the various options, and why anything involving user files is less effective than "nuking from orbit".

Depending on the issue, you may want to add "doing nothing" or "waiting" that will inform your management of the risks to the business at large.

Answer 3

You can drink all the red wine anti-virus you want to try and prevent getting cancer, but once you get that first tumor, more drinking isn't going to help. You need to cut it out and make sure that you get all of it, because if you don't it will come back again.

Once you get infected with a virus, the obvious symptoms are an annoyance, but it is what you cannot see where the true danger lies. Backdoors, rootkits, and botnets can all hide without any indication that there is anything wrong. Sometimes the hidden dangers are combined with obvious dangers so you feel secure once the obvious symptoms are gone, but the obvious is a distraction from the hidden.

Once you know that you have been infected, you do not know how far the infection goes, and not knowing that means you do not know what is at risk. The most basic course of action is to nuke it from orbit. That way you know where you are and you know what your risk is, even if there is a significant cost to starting over from scratch.

Answer 4

That's easy - just finish the quote in your question, from Aliens.

It's the only way to be sure.

That's really all there is to it. Nothing more, nothing less. Let them know that if you run the AV software on it and the software says it has found and removed the virus, then maybe they're ok. Maybe. If the virus was really removed. If that was really the only virus.

To respond to what someone else posted about "How to save user data from the machine" the answer is that you don't. "TAKE OFF AND NUKE THE ENTIRE SITE FROM ORBIT" That means you restore from backup and they lose anything that wasn't backed up. It's not the easy thing to do, it's the right thing to do.

Because it's the only way to be sure.

Answer 5

Try spies. The last James Bond opus appears to make millions of entries, so the crowd at large is, for now, receptive to spy stories. Explain that once unreliable/hostile people are in charge (that's the "compromised" setup), there is no way to recover proper security by asking them to do it; and yet, that's what running an AV on an infected machine is about. Spy networks around the world have always been seggregated into autonomous cells precisely so that the rotten parts can be severed. Once an agent has been subverted, you can perhaps subvert him back, but never will you trust him again.

To make the demonstration more complete, talk about infected keyboard firmwares, which highlights the necessity to really put the machine to fire. Reusing the hardware, even after a wipeout, is risky. Therefore, managers/users should feel grateful that we accept not to do the full cleansing, and limit ourselves to logical nuking, not physical. Make it felt that it already is a grave compromission on your part.

Answer 6

To be the devil's advocate, management has heard all of this. Security is risk management and they need to make the decision. Just remind them of the tools.

Risk = Probability of occurance X impact of occurance

100% chance of production downtime during the rebuild and costs of rebuild vs. 0.01% chance of malicious software or backdoors remaining in the system.

Who would control the back doors? Cybercriminals performing serial crimes from China, Russia or Eastern Europe. What would they have access to? system data, file shares, keyboard sniffers, microphones, cameras, etc. How much would they be able to sell that information for? How long could they go undetected?

What does that mean for the machine in question and the information on it?

Then provide your assessment of the target (using your limited information), and let them make the decision. They know the finances and they have more insight as to the true value of the target.

There are plenty of other vectors for attack. Disgruntled employees, contractors and their equipment, backdoors in legit software, misconfigured security systems, unencrypted drives, etc. It's an imperfect world. The money and time spent rebuliding might be better spent elsewhere, like fixing the password policy, hardening your email servers, improving backups, etc.

Answer 7

From the technician's point of view, you're certainly correct. But the CEO is not looking at this from the technician's point of view. So you either have to make the argument in terms that make sense to him.

No solution is absolutely 100% effective. Not even "nuke it from orbit". What you get is probabilities. Put that on a cost-benefit table, and you're speaking language the CEO can understand: (numbers here are examples only)

• So if I clean off the obvious malware alone, then the cost is lowest (1 hour labor/downtime) and perhaps that 20% effective (80% of the time, the attacker will quickly return).

• If I clean off the obvious malware and spend extra time examining any files changed in the past 48 hours, then I get a higher success rate and higher cost: 6 hours labor/downtime, 60% success rate

• Perhaps if I do all the above plus reinstall all the system packages (e.g. on RH systems: yum reinstall openssh-server etc), then the cost and success rates go higher: 12 hours labor/downtime, 95% success rate

• If I do all the above, plus spend additional time checking/removing any files in /bin, /sbin/ etc., that are not owned by any package, then perhaps that adds another 4 hours, and give me an extra 3% points on my success rate.

• Finally, if I "nuke it from orbit", I get the highest cost and success rate: 48 hours labor/downtime, 99.995% success rate

Then from there, we figure out how much each hour of work/downtime costs, plus add the per-incident cost of each exploit, and we start to get an idea of which solution will likely cost the least/most in the long term. And now it's a simple business decision. CEOs are good at that.

Of course, the solutions listed above assume a *NIX environment, but you could come up with a similar list for Windows systems. Make sure you throw AV in there as an option with a realistic associated success rate.

Here's the rub: the probabilities are hard to come up with. Unless you've done or seen a lot of these half-way solutions, you probably don't have any basis to go on. Plus convincing a security professional to go along with solution 3 above when he knows that he's specifically ignoring some potentially serious threats is going to be a tough sell.

But the decision and the risk are the responsibility of the CEO, not the security professional. He may decide to go with a less secure option, but as long as he knows what risks he's taking, then he should be free to do so.

Answer 8

That's a tough one. You have to use concepts that the average person will understand, and find a way to make them care. I would use biological viruses and how they work to explain how computer viruses work because it's something everybody has experience with, and has a potential to get the user to be "sympathetic" to the computer's situation.

A biological virus subverts a cell, making it do what the virus wants. The virus becomes essentially a zombie. You can't trust the cell to do what it is supposed to do, and you can't stop the virus-infected cell and make it normal again, the machinery has been taken control of so thoroughly you can only kill it.

Older computer virus didn't mimic biological viruses very closely. Their level of sophistication was such that they were able to do some things, but not infect systems to to the level they couldn't be removed. Their survival depended more on there not being an AV on the system.

Now computer viruses mimic biological ones much more closely, they are able to subvert a system so thoroughly that you can never be sure they are clear. It may say it's clear, but the virus is so totally in control it can prevent an AV from detecting it. The computer's like a zombie cell, and the only way to prevent the virus from spreading is to kill it.

Answer 9

Imagine we're living in a horror movie. Your fiancé or fiancée (as the case may be) has been cursed by a witch and is now spewing projectile vomit while spinning their head unnaturally.

You, as an exorcist and amateur brain surgeon, have two options:

1. Cast out the demons totally and restore your beloved's soul to ownership of their body.
2. Attempt days-long and delicate brain surgery on a supernaturally strong and dangerous body that will fight your every attempt.

Option 1 is simple, cheap, and works (in horror movies).

Option 2 requires teams of highly trained and expensive doctors, and probably won't work because you can't anaesthetize demons.

---

Restoring software from backup is analogous to option 1, and unlike exorcism works IRL.

Option 2 is analogous to employing sysadmins to check program binaries, data files, and configurations against a known good copy that they could just have installed on the machine in the first place.

Answer 10

Its worth pointing out, that it is usually ok to transfer valuable non-executable data (with no recent backup) from an infected machine, before nuking from orbit (wiping hard drive, reinstall OS from a safe source). Stuff like plaintext docs (e.g., latex manuscript or source code) or important media files (e.g., images of family vacation) may be worth recovering if there is no recent backup. However, you need to be suspicious that the virus let an attacker have full control of the infected system, and the attacker may have modified your data. This could include introduce backdoors into your source code, creating their own admin users in your databases, altering configuration files so the system is in a weak configuration that can be attacked again, etc. (I would read through all source code with a fine toothed comb to make sure no subtle changes were made -- and that's only if it is not security critical). Also be careful of some media files potentially contain viruses, e.g., MS Office documents with macro viruses (this case best, I'd export the copy the text content out of the .doc/.xls into a plain-text file from the infected system when its not connected to anything). Also be careful transferring the data off the infected machine (to not reinfect the other machine); e.g., I'd probably do something like boot off a linux live cd, mount the infected hard drive with -noexec, connect to the internet, and selectively copy important files, and if possible try comparing them to the most recent backup.

The reason for nuking from orbit is its the only way you can have any confidence to safely use that computer again. Anti-virus software works by identifying known malware and as such cannot do so with 100% accuracy (and anti-virus software running on an infected computer may have been tampered with by the virus significantly lowering its chances of completely removing the virus). Starting back at a safe point means you won't have your valuable data get stolen, or have to repeat the process again in a week (possibly on more PCs as the infection spread). Re-installation can be automated and takes under a day; roughly an equivalent time to run a full virus scan that has no guarantee of effectiveness. If downtime is an issue, there needs to be redundancy in the amount of computing resources available to the organization.