If you have large distributed network that is known as being infected with viruses and etc.. and you cannot format the computers becuase its large scale. What are the best ways to handle this situation from the cso viewoint?

  • 1
    How large? (Hundreds of machines? Thousands?) Elaborate on why re-building isn't feasible - there are many methods for bulk re-imaging, usually involving PXE (e.g. symantec.com/deployment-solution?fid=endpoint-management) – scuzzy-delta Jul 4 '13 at 21:54
  • It also has low bandwitch to deploy from the net.its about hundred that seperate physically from each other – user1387682 Jul 4 '13 at 22:02
  • 2
    Define "Distributed network" in this context. What you seem to be describing is the equivalent of remotely managing 100 individual user's machines. – schroeder Jul 4 '13 at 22:49
  • Yeah, that's what it sounds like to me also. 100% BYOD. I hope our friend likes challenges. – scuzzy-delta Jul 4 '13 at 23:02

The SANS course that includes incident handling, introduced at https://www.sans.org/course/hacker-techniques-exploits-incident-handling recommends the following steps:

  1. Preparation
  2. Identification
  3. Containment
  4. Eradication
  5. Recovery
  6. Lessons Learned

Obviously, it's too late for step 1...

A network IDS (eg. snort) watching all internet traffic will help you identify infected hosts (#2). It will also help you identify where you're being attacked from so you can filter those IPs and networks at your firewall (#3). Don't forget that most control channels will originate from your machines out to the attackers' servers, rendering basic firewall rulesets ineffective. The ability to watch intra-host traffic as well as external traffic at the NIDS will help spot lateral transfers where an infected host directly compromises another.

As infected hosts are identified, rebuild them (#4 & 5). Use a unique local administrator password for each one, saving them offline (eg. a notebook in a safe or locked room) for use next time that host is compromised. That way, you're not exposing domain admin credentials when logging into a compromised host, and local admin on one host does not equal local admin on all of them.

Full packet capture will allow you to review malicious activities, extract malware, droppers, exfiltrations, and such. With limited network bandwidth, hardware requirements for capturing all traffic drop to levels where commodity hardware can handle it.

If your infrastructure is Windows, radically limit your domain admin accounts and rotate their passwords often, especially when used to access a compromised host. Log all network accesses, collect your logs centrally, and review them regularly. For SSH access, use PKI, generate new private keys and remove old keys from authorized_hosts. Change any local account passwords on all hosts and don't reuse passwords on multiple hosts.

It sounds like you may be over your head on this. It may be wise to hire an outside firm to help you identify which hosts to clean, IPs to blackhole, etc. Search for incident handling services and shop around.

| improve this answer | |

The basics are the basics:

  • contain infected machines
  • reinstall/re-image infected machines
  • change all passwords
  • set up better detection for faster response in the future

This is the best way. If you have cost constraints for containing and reinstalling/re-imaging, then that might be the 'real' question and therefore we would need more data to help.

From a CSO point of view: if you don't have physical control over the computers, then they aren't your computers anymore, and you should rethink your responsibility over them. If you do truly have responsibility for these machines, but do not have the ability to remotely "nuke them from orbit", then you need to start working on doing that ASAP.

If that level of control is not possible, then you need to start thinking about changing your expectations to allow your machines to be permanently infected. This could actually be a cheaper option, but it might likely require a total rethink on your infrastructure and trust relationships between the machines. For instance, think of each of your employees as if they are merely members of the general public.

| improve this answer | |

So the scenario is: About a hundred machines, all in separate locations, perhaps interconnected via a VPN. No ability to deploy an image by PXE.

Given the constraints, it seems the only option is to slowly contain and rehabilitate. Create a new VLAN/VPN. Run Antivirus software configured for comprehensive scanning. Machines that are "clean" go into the new VPN. Continue until all machines are clean.

This will be very difficult and time-consuming, and poses a high risk of re-infection because you can't detect 100% of malware 100% of the time.

Once the crisis is over, the organization needs to develop and implement a management plan (with appropriate budget!). It's 2013 - the days of an entire network being infected should be well behind us.

| improve this answer | |
  • thank you.what about investigation of what the impact of the malware?is there tools like ids that could help? – user1387682 Jul 4 '13 at 22:24
  • 1
    Yes, an IDS would help to identify the malware traffic, but where would you deploy the IDS? – schroeder Jul 4 '13 at 22:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.