If you have large distributed network that is known as being infected with viruses and etc.. and you cannot format the computers becuase its large scale. What are the best ways to handle this situation from the cso viewoint?
The SANS course that includes incident handling, introduced at https://www.sans.org/course/hacker-techniques-exploits-incident-handling recommends the following steps:
- Lessons Learned
Obviously, it's too late for step 1...
A network IDS (eg. snort) watching all internet traffic will help you identify infected hosts (#2). It will also help you identify where you're being attacked from so you can filter those IPs and networks at your firewall (#3). Don't forget that most control channels will originate from your machines out to the attackers' servers, rendering basic firewall rulesets ineffective. The ability to watch intra-host traffic as well as external traffic at the NIDS will help spot lateral transfers where an infected host directly compromises another.
As infected hosts are identified, rebuild them (#4 & 5). Use a unique local administrator password for each one, saving them offline (eg. a notebook in a safe or locked room) for use next time that host is compromised. That way, you're not exposing domain admin credentials when logging into a compromised host, and local admin on one host does not equal local admin on all of them.
Full packet capture will allow you to review malicious activities, extract malware, droppers, exfiltrations, and such. With limited network bandwidth, hardware requirements for capturing all traffic drop to levels where commodity hardware can handle it.
If your infrastructure is Windows, radically limit your domain admin accounts and rotate their passwords often, especially when used to access a compromised host. Log all network accesses, collect your logs centrally, and review them regularly. For SSH access, use PKI, generate new private keys and remove old keys from authorized_hosts. Change any local account passwords on all hosts and don't reuse passwords on multiple hosts.
It sounds like you may be over your head on this. It may be wise to hire an outside firm to help you identify which hosts to clean, IPs to blackhole, etc. Search for incident handling services and shop around.
The basics are the basics:
- contain infected machines
- reinstall/re-image infected machines
- change all passwords
- set up better detection for faster response in the future
This is the best way. If you have cost constraints for containing and reinstalling/re-imaging, then that might be the 'real' question and therefore we would need more data to help.
From a CSO point of view: if you don't have physical control over the computers, then they aren't your computers anymore, and you should rethink your responsibility over them. If you do truly have responsibility for these machines, but do not have the ability to remotely "nuke them from orbit", then you need to start working on doing that ASAP.
If that level of control is not possible, then you need to start thinking about changing your expectations to allow your machines to be permanently infected. This could actually be a cheaper option, but it might likely require a total rethink on your infrastructure and trust relationships between the machines. For instance, think of each of your employees as if they are merely members of the general public.
So the scenario is: About a hundred machines, all in separate locations, perhaps interconnected via a VPN. No ability to deploy an image by PXE.
Given the constraints, it seems the only option is to slowly contain and rehabilitate. Create a new VLAN/VPN. Run Antivirus software configured for comprehensive scanning. Machines that are "clean" go into the new VPN. Continue until all machines are clean.
This will be very difficult and time-consuming, and poses a high risk of re-infection because you can't detect 100% of malware 100% of the time.
Once the crisis is over, the organization needs to develop and implement a management plan (with appropriate budget!). It's 2013 - the days of an entire network being infected should be well behind us.