Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

When a machine has been infected with malware, most of us here immediately identify the appropriate action as "nuke it from orbit" - i.e. wipe the system and start over. Unfortunately, this is often costly for a company, especially if backups are configured in a less-than-optimal fashion. This can produce resistance from management and users who just want to carry on using the machine for work. After all, as far as they're concerned, they can "just run AV over it" and everything will be fine.

How do you explain the problem to management and users of a non-technical nature? I could easily produce a technical reason, but I'm having trouble coming up with the appropriate wording for non-technical users. I'd especially appreciate any way of speaking that the recipient can identify with, e.g. appealing to a manager's sense of risk management.

share|improve this question
1  
More to the point. How can a user's data safely be transferred to a new machine. This is a problem I run into. –  k to the z Nov 19 '12 at 16:17
1  
@ktothez It's better to not transfer user data; e.g., rely on a safe backup from before infection. However, if necessary I'd do something like my answer convert MS Office docs to plaintext docs/CSV files on the infected machine, boot into a linux live cd on the infected computer, mount the infected hard drive with -noexec, and then selectively copy the limited number of plain-text documents from the infected machine to another computer. However, be suspicious of your files being maliciously altered--e.g., config files, passwords, data, etc. –  dr jimbob Nov 19 '12 at 20:15
    
Is it not enough merely to say that the course of action you propose is the only one which guarantees the desired outcome (assuming of course that it is the only course of action which guarantees the desired outcome)? –  jah Nov 19 '12 at 22:10
4  
@ jah unfortunately IT doesn't run the show most places. IT decisions are often overridden by people lacking IT knowledge. –  k to the z Nov 19 '12 at 22:29
9  
@jah Sadly, no. The desired outcome for them is absolutely minimal expenditure in order to mitigate the risk. The desired outcome for me is absolutely minimal risk. The trick is to find a balance that works well for both sides of the coin. Unfortunately, that balance can't easily be reached unless both sides understand each other. –  Polynomial Nov 19 '12 at 22:53

10 Answers 10

up vote 71 down vote accepted

In my experience management doesn't like to listen to clever analogies. Depending on the person they care about the bottom line in dollars or hours of productivity. I would explain "The actual bottom line is that a compromise of our data will cost the company approximately X dollars + X hours to recover. This is X% likely to happen given the malware that is on this machine. A new install will cost X dollars + X hours to recover. You pick the appropriate action."

It's short and clear and doesn't really leave them any room to argue. They will clearly understand the risk and should make the right decision.

share|improve this answer
33  
The problem is the fundamental uncertainty regarding X% - once an infection has occurred, it could be anywhere between 0% and 100%. There's no way to tell where, and there's only one way to get it down to 0%. This is a more fiddly concept than risk. Also, for clarity, I'd suggest using different variables for dollars and hours, e.g. Y and Z. –  user3490 Nov 19 '12 at 17:03
4  
Good answer, KDEx: security is an economic good, with a cost and a value. My own thought is that the X% of malware is probably wildly overestimated, leading the the "nuke it from orbit" habit, but I've only really dealt with Solaris and Linux malware. Windows malware certainly sounds weirder and more persistent. –  Bruce Ediger Nov 19 '12 at 19:09
1  
The problem is in costing the effect of downtime - but that shouldn't be your responsibility - however you do need to know that to cost the potential solutions. But do beware if relying on AV as a fix: Art Taylor estimates a 30% likelihood of a good outcome - but IMHO this is rather optimistic - you'll likely end up just temporarily removing some of the symptoms. –  symcbean Nov 19 '12 at 21:49
    
@KDEx you need to consider your audience and tailor your answer to them. $X > $Y can be the best answer for certain audiences, but not all managers think so concretely, even having the numbers laid out. If they did, they would not be trying to override the expert in the situation with their own technical solution (run AV over it!). Granted, you need to know your numbers when the discussion gets to this point, but even you state there is room for interpretation (X%), which means there is always room to argue. –  schroeder Nov 19 '12 at 22:28
1  
@Polynomial that sounds like a lot of uncertainty...which to many in management (depending on industry) is the definition of risk. Therefore by definition it's a hard estimate. Being an expert in the field you are the most well positioned person to make that assessment and provide it to the decision makers, no? –  KDEx Nov 20 '12 at 12:53

That's a tough one. You have to use concepts that the average person will understand, and find a way to make them care. I would use biological viruses and how they work to explain how computer viruses work because it's something everybody has experience with, and has a potential to get the user to be "sympathetic" to the computer's situation.

A biological virus subverts a cell, making it do what the virus wants. The virus becomes essentially a zombie. You can't trust the cell to do what it is supposed to do, and you can't stop the virus-infected cell and make it normal again, the machinery has been taken control of so thoroughly you can only kill it.

Older computer virus didn't mimic biological viruses very closely. Their level of sophistication was such that they were able to do some things, but not infect systems to to the level they couldn't be removed. Their survival depended more on there not being an AV on the system.

Now computer viruses mimic biological ones much more closely, they are able to subvert a system so thoroughly that you can never be sure they are clear. It may say it's clear, but the virus is so totally in control it can prevent an AV from detecting it. The computer's like a zombie cell, and the only way to prevent the virus from spreading is to kill it.

share|improve this answer

"You can drink all the red wine anti-virus you want to try and prevent getting cancer, but once you get that first tumor, more drinking isn't going to help. You need to cut it out and make sure that you get all of it, because if you don't it will come back again."

"Once you get infected with a virus, the obvious symptoms are an annoyance, but it is what you cannot see where the true danger lies. Backdoors, rootkits, and botnets can all hide without any indication that there is anything wrong. Sometimes the hidden dangers are combined with obvious dangers so you feel secure once the obvious symptoms are gone, but the obvious is a distraction from the hidden."

"Once you know that you have been infected, you do not know how far the infection goes, and not knowing that means you do not know what is at risk. The most basic course of action is to nuke it from orbit. That way you know where you are and you know what your risk is, even if there is a significant cost to starting over from scratch."

share|improve this answer
34  
Your cancer analogy doesn't hold up well. "Nuke it from orbit" corresponds to "shoot the cancer victim and start with his most recent clone", not the preferred solution generally. –  CodesInChaos Nov 19 '12 at 15:24
5  
@CodesInChaos, that entirely depends on the user in question. ;) –  GdD Nov 19 '12 at 15:24
3  
It holds up fine. We can't replace all cells in a human body, but we wish we could. Cancer is never 'cured' it is 'survived' because we can never be sure all cancer cells are removed. With a computer, we have the ability to replace all the cells to ensure that all malignant elements are no longer a factor. –  schroeder Nov 19 '12 at 15:30
1  
I'd say iodine rather than red wine, but I do like that cancer analogy. Needing to cut it out of the network works quite nicely. –  Polynomial Nov 19 '12 at 15:48

Imagine we're living in a horror movie. Your fiancé or fiancée (as the case may be) has been cursed by a witch and is now spewing projectile vomit while spinning their head unnaturally.

You, as an exorcist and amateur brain surgeon, have two options:

  1. Cast out the demons totally and restore your beloved's soul to ownership of their body.
  2. Attempt days-long and delicate brain surgery on a supernaturally strong and dangerous body that will fight your every attempt.

Option 1 is simple, cheap, and works (in horror movies).

Option 2 requires teams of highly trained and expensive doctors, and probably won't work because you can't anaesthetize demons.


Restoring software from backup is analogous to option 1, and unlike exorcism works IRL.

Option 2 is analogous to employing sysadmins to check program binaries, data files, and configurations against a known good copy that they could just have installed on the machine in the first place.

share|improve this answer
    
I think you've misunderstood. I can provide a technical explanation just fine, but I'm looking for a way to explain it to a non-technical user or manager. Also, 100MB is nothing compared to a typical OS installation these days - you're looking at tens of gigabytes for Windows 7 or 8. –  Polynomial Nov 19 '12 at 15:40
    
@Polynomial, this was my attempt to provide an explanation at a level of detail that could be grasped by non-technical user.s Yes, 100's of MBs is a slim install for a user-facing OS these days. Many special purpose servers though still run slimmer OSes. –  Mike Samuel Nov 19 '12 at 15:44
    
@Polynomial please see my edits. –  Mike Samuel Nov 19 '12 at 16:00
3  
I would personally have put a zombie slant on this - if a fellow survivor is bitten, do you a) apply first aid, and let them shrug it off, or do you b) kill them right there to prevent them from harming anyone else, and burn their corpse afterwards? –  user3490 Nov 19 '12 at 16:50

Try spies. The last James Bond opus appears to make millions of entries, so the crowd at large is, for now, receptive to spy stories. Explain that once unreliable/hostile people are in charge (that's the "compromised" setup), there is no way to recover proper security by asking them to do it; and yet, that's what running an AV on an infected machine is about. Spy networks around the world have always been seggregated into autonomous cells precisely so that the rotten parts can be severed. Once an agent has been subverted, you can perhaps subvert him back, but never will you trust him again.

To make the demonstration more complete, talk about infected keyboard firmwares, which highlights the necessity to really put the machine to fire. Reusing the hardware, even after a wipeout, is risky. Therefore, managers/users should feel grateful that we accept not to do the full cleansing, and limit ourselves to logical nuking, not physical. Make it felt that it already is a grave compromission on your part.

share|improve this answer
1  
Whilst this is a good way of putting it from a literary perspective, I fear that such an explanation would leave most middle-management dumbfounded from a completely different point of view. –  Polynomial Nov 19 '12 at 15:50
    
Do you know if Apple did anything about the keyboard firmware hack, or if it's been seen in the wild? –  Dan Neely Nov 19 '12 at 16:39
    
I am not aware of anything done against it, or of any sigthing in the wild. But this does not prove much... –  Thomas Pornin Nov 19 '12 at 18:03

I would avoid the biological or non-business analogies (unless this is a hospital). Your job is to assess risk, cost, and provide options. Your management's job is to make the decision based on your analysis and advice.

Generally, an approach in a tabular format is best. "approach", "likelihood of correcting the problem", "cost" are the minimum needed. You can call the second "Vegas" if you absolutely have to get cute.

For example, in this case, you may have the following.

Approach                       Prognosis     Cost
Run anti-virus on machine      30%           4 hours IT, 4 hours downtime
Replace machine w/new machine  75%           $3,000, 16 hours IT, 4 hours downtime
AV machine, copy user files, 
    replace machine, restore   60%           $3,000, 24 hours IT, 4 hours user, 8 hours
    files                                        downtime

In this list (assuming a user desktop), the real problem is user behavior. You'll want to document why the prognosis is < 100% for the various options, and why anything involving user files is less effective than "nuking from orbit".

Depending on the issue, you may want to add "doing nothing" or "waiting" that will inform your management of the risks to the business at large.

share|improve this answer
4  
+1 for the cost-benefit table. This is really what most management wants to see. –  Cypher Nov 19 '12 at 19:07
1  
This is a very good answer. The table of course, and also the scenarios of doing nothing or waiting that you mentioned, even if you can't quantify the downside risk that well. They are nevertheless viable options, and that's great that you included them. Context specific analogies are great but only if directly relevant to the business. If you can't do it with certainty, best not to try, and definitely better not to frame things in terms that will reinforce tedious stereotypes about IT and security staff work culture. –  Feral Oink Nov 19 '12 at 20:07
    
Interesting answer, the downside in my opinion with a hard numbers answer is where do the percentage likelihoods come from? In a lot of security when people try to quantify things like probabilities and costs the numbers have an element of "best guess" as there really is no good way to predict them and then if challenged on why they are that level, it's hard to defend them.. –  Rоry McCune Nov 19 '12 at 22:07
    
@RoryMcCune While I don't know if it's what ArtTaylor has in mind, I would think of it along the lines of IT saying "This is what we want you to do"... While still leaving the final decision in their hands –  Izkata Nov 20 '12 at 19:15
    
Why would you replace the machine? Why wouldn't you just wipe the hard drive and reinstall the OS? –  naught101 Nov 21 '12 at 9:40

From the technician's point of view, you're certainly correct. But the CEO is not looking at this from the technician's point of view. So you either have to make the argument in terms that make sense to him.

No solution is absolutely 100% effective. Not even "nuke it from orbit". What you get is probabilities. Put that on a cost-benefit table, and you're speaking language the CEO can understand: (numbers here are examples only)

  • So if I clean off the obvious malware alone, then the cost is lowest (1 hour labor/downtime) and perhaps that 20% effective (80% of the time, the attacker will quickly return).

  • If I clean off the obvious malware and spend extra time examining any files changed in the past 48 hours, then I get a higher success rate and higher cost: 6 hours labor/downtime, 60% success rate

  • Perhaps if I do all the above plus reinstall all the system packages (e.g. on RH systems: yum reinstall openssh-server etc), then the cost and success rates go higher: 12 hours labor/downtime, 95% success rate

  • If I do all the above, plus spend additional time checking/removing any files in /bin, /sbin/ etc., that are not owned by any package, then perhaps that adds another 4 hours, and give me an extra 3% points on my success rate.

  • Finally, if I "nuke it from orbit", I get the highest cost and success rate: 48 hours labor/downtime, 99.995% success rate

Then from there, we figure out how much each hour of work/downtime costs, plus add the per-incident cost of each exploit, and we start to get an idea of which solution will likely cost the least/most in the long term. And now it's a simple business decision. CEOs are good at that.

Of course, the solutions listed above assume a *NIX environment, but you could come up with a similar list for Windows systems. Make sure you throw AV in there as an option with a realistic associated success rate.

Here's the rub: the probabilities are hard to come up with. Unless you've done or seen a lot of these half-way solutions, you probably don't have any basis to go on. Plus convincing a security professional to go along with solution 3 above when he knows that he's specifically ignoring some potentially serious threats is going to be a tough sell.

But the decision and the risk are the responsibility of the CEO, not the security professional. He may decide to go with a less secure option, but as long as he knows what risks he's taking, then he should be free to do so.

share|improve this answer
    
The real problem is that when the CEO decides on something less than nuking from orbit and the infection ends up surviving and doing serious damage the suit is liable to turn around and say that since the option done was 95% effective and failed that 95% of the blame goes to the sysadmin who did the recovery and since someone needs to be sacked as a scapegoat for the fiasco it should obviously be the person who failed to successfully clean the infection. –  Dan Neely Nov 19 '12 at 18:49
    
@DanNeely That's certainly the sort of thing you'd expect to worry about. But as long everything is properly explained and documented, it shouldn't be. If purchase a product with a documented 2% defect rate in your contract, you can't lay 98% of the blame of the defects on the manufacturer. It's 100% the decision maker. If you are in a workplace where this is a concern, you should simply (a) flatly refuse to do what the CEO tells you to do, and/or (b) find a new job. –  tylerl Nov 19 '12 at 18:56
    
yum reinstall openssh-server etc depends on your being able to trust yum to do what it is supposed to. Do you? –  Michael Kjörling Nov 19 '12 at 19:45
1  
@MichaelKjörling typically, yes. Again, we're playing on probabilities and averages. ssh/sshd are frequently trojaned, as are coreutils including md5sum. But so far on the thousands of hacked systems I've examined, they haven't trojaned yum or python or rpm. It's not impossible, it's just not popular (yet). –  tylerl Nov 19 '12 at 20:27

To be the devil's advocate, management has heard all of this. Security is risk management and they need to make the decision. Just remind them of the tools.

Risk = Probability of occurance X impact of occurance

100% chance of production downtime during the rebuild and costs of rebuild vs. 0.01% chance of malicious software or backdoors remaining in the system.

Who would control the back doors? Cybercriminals performing serial crimes from China, Russia or Eastern Europe. What would they have access to? system data, file shares, keyboard sniffers, microphones, cameras, etc. How much would they be able to sell that information for? How long could they go undetected?

What does that mean for the machine in question and the information on it?

Then provide your assessment of the target (using your limited information), and let them make the decision. They know the finances and they have more insight as to the true value of the target.

There are plenty of other vectors for attack. Disgruntled employees, contractors and their equipment, backdoors in legit software, misconfigured security systems, unencrypted drives, etc. It's an imperfect world. The money and time spent rebuliding might be better spent elsewhere, like fixing the password policy, hardening your email servers, improving backups, etc.

share|improve this answer

That's easy - just finish the quote in your question, from Aliens.

It's the only way to be sure.

That's really all there is to it. Nothing more, nothing less. Let them know that if you run the AV software on it and the software says it has found and removed the virus, then maybe they're ok. Maybe. If the virus was really removed. If that was really the only virus.

To respond to what someone else posted about "How to save user data from the machine" the answer is that you don't. "TAKE OFF AND NUKE THE ENTIRE SITE FROM ORBIT" That means you restore from backup and they lose anything that wasn't backed up. It's not the easy thing to do, it's the right thing to do.

Because it's the only way to be sure.

share|improve this answer
1  
Hold on a second. This installation has a substantial dollar value attached to it. –  CaptainMurphy Nov 19 '12 at 22:33
    
I can bill them! –  Mark Allen Dec 6 '13 at 0:09

Its worth pointing out, that it is usually ok to transfer valuable non-executable data (with no recent backup) from an infected machine, before nuking from orbit (wiping hard drive, reinstall OS from a safe source). Stuff like plaintext docs (e.g., latex manuscript or source code) or important media files (e.g., images of family vacation) may be worth recovering if there is no recent backup. However, you need to be suspicious that the virus let an attacker have full control of the infected system, and the attacker may have modified your data. This could include introduce backdoors into your source code, creating their own admin users in your databases, altering configuration files so the system is in a weak configuration that can be attacked again, etc. (I would read through all source code with a fine toothed comb to make sure no subtle changes were made -- and that's only if it is not security critical). Also be careful of some media files potentially contain viruses, e.g., MS Office documents with macro viruses (this case best, I'd export the copy the text content out of the .doc/.xls into a plain-text file from the infected system when its not connected to anything). Also be careful transferring the data off the infected machine (to not reinfect the other machine); e.g., I'd probably do something like boot off a linux live cd, mount the infected hard drive with -noexec, connect to the internet, and selectively copy important files, and if possible try comparing them to the most recent backup.

The reason for nuking from orbit is its the only way you can have any confidence to safely use that computer again. Anti-virus software works by identifying known malware and as such cannot do so with 100% accuracy (and anti-virus software running on an infected computer may have been tampered with by the virus significantly lowering its chances of completely removing the virus). Starting back at a safe point means you won't have your valuable data get stolen, or have to repeat the process again in a week (possibly on more PCs as the infection spread). Re-installation can be automated and takes under a day; roughly an equivalent time to run a full virus scan that has no guarantee of effectiveness. If downtime is an issue, there needs to be redundancy in the amount of computing resources available to the organization.

share|improve this answer
    
And then there are the fonts/images/... that cause buffer overrun attacks in the viewers or the PDFs that break out of the viewers - you can never be sure if the system hasn't been hit by a targeted 0-day caused by some data file. –  Martin Schröder Nov 22 '12 at 23:29
    
@MartinSchröder - I agree that pdfs (like *.doc) are dangerous as they aren't simple plaintext, but will often auto-execute embedded scripts. I disagree with images which will generally be safe when viewed through mature viewers. In principle, image viewers could be susceptible to buffer-overflow attacks, but only if poorly written (e.g., unsafe language with unsafe libraries never doing bounds checking). I'll browse the web and view whatever images are presented with no danger of viruses (unlike pdf/doc). –  dr jimbob Nov 26 '12 at 17:23

protected by Rory Alsop Nov 19 '12 at 21:56

Thank you for your interest in this question. Because it has attracted low-quality answers, posting an answer now requires 10 reputation on this site.

Would you like to answer one of these unanswered questions instead?

Not the answer you're looking for? Browse other questions tagged or ask your own question.