What can a company do against insiders going rogue and negatively affecting essential infrastructure?

Question

In 2013, a Citibank employee had a bad performance review that ticked him off. The results were devastating:

Specifically, at approximately 6:03 p.m. that evening, Brown knowingly transmitted a code and command to 10 core Citibank Global Control Center routers, and by transmitting that code, erased the running configuration files in nine of the routers, resulting in a loss of connectivity to approximately 90 percent of all Citibank networks across North America.

Now, there is a question about securing a network against attacks from the inside, but that question explicitly excludes insiders going rogue. There is also a question about protecting a database against insiders, but that's concerning high-tier problems.

I also read What is the procedure to follow against a security breach?, but most answers on there act based on the insider being an employee that got fired. I'm asking about someone who hasn't been fired yet. They might have had a poor performance review, but haven't yet been terminated yet. They might just be unhappy about something their partner did, or they might have gotten upset about something.

The problem I'm describing here is a large company where a user who is unhappy about their job snaps on a certain day and issues system-breaking commands tha they have full privileges to issue. Things like wiping machines, physically damaging essential infrastructure,... purely technical interference, nothing like leaking emails or secrets. The aim is just to do as much damage as possible to the infrastructure to go out with a bang.

The article gives a few cursory mentions of things to do, but nothing really concrete. What things can be done to prevent sudden rogue insiders from negatively impacting essential infrastructure using techniques they're privileged to do?

Answer 1

Two-man rule - configure your systems so that all privileged access requires two people.

This could be a physical control - privileged access can only come from the NOC, and inside the NOC people physically enforce the rule.

More practical would be a scripting system. Sys-admins don't directly have root access, but they can submit scripts to be run as root. They will only be run after a separate person has reviewed and approved the script. There would still need to be a method for SSH access in an emergency - and the two-man rule could be maintained in that case using physical controls.

The NSA implemented this after the Snowden leaks. I have never seen a full two-man system in any of the commercial or government systems I have audited - although I have seen various partial attempts.

Update - there's more information on how to implement this on a separate question.

Answer 2

What things can be done to prevent sudden rogue insiders from negatively impacting essential infrastructure using techniques they're privileged to do?

In practice, very little. But to explain why, let me talk about what you can do.

The issue here is that the user is "privileged" - they have been granted the power legitimately.

There are some things that can be done to limit the power given to legitimate users, even privileged administrators:

• Control over available commands using something like sudo or PowerBroker.
• Dual control (the "two-man rule" @paj28 describes
• Workflow controls (which are often a form of dual control)

Now, these controls are used far less than they could be. Why? Because privileged users are trusted by definition. So I say very little not because there are no controls, but because the cost-benefit ratio of such controls when applied to trusted personnel is not enough to justify it.

Also note that the attack vector here was "in the plumbing" - if Citibank has dual controls, they're probably focused on things like funds transfers, whereas this attack came in at the knees and just took the underlying network down. These vital-but-quiet systems often have smaller circles of privileged users and less excessive controls.

The real failure here was not that there were not technical controls, but that the personnel controls failed miserably. It is standard practice to revoke access of privileged employees before they are terminated. Whoever decided that no such precaution was necessary when introducing conflict with a privileged employee used poor judgement.

(The company also employed punitive controls - the attacker is now sentenced to almost 2 years in prison and must pay nearly $80k. As the article points out, those things don't fix any of this.)

Answer 3

One approach is to accept that rogue actions cannot be prevented and focus on making sure the damage can be repaired. For example, make sure the routers have a separate control plane via which they can be brought back online. Make sure you have read-only backups (e.g. off-site tapes), so if someone wipes out all hard drives you can recover the data. Make sure data and code can be rolled back to a known good state quickly.

These safeguards will also help a lot in the case of unintentional mistakes.

Answer 4

Audit. In particular network traffic and performed actions/operations on particular machines. You want to capture, who did what, when they did it and from where. Whilst this wont prevent an attack, it will help deter such actions if the insider believes that they will be identified and caught.

Then you have to get into the issue of tamper-proof auditing mechanisms

Answer 5

The question of protecting a system or network from an insider, most specifically from the people who's own job description includes creating and managing such system has always been a tricky one.

First, what one must understand is that, in the end, it is fully impossible to prevent all kinds of attack against an infrastructure from the inside, because that would imply restricting all contact with the infrastructure, making it particularly useless.

However, there are ways we can prevent and minimize any damage to the system. To this process, I personally recognize there are three stages:

1. The Two-Man rule
2. The Accountability rule
3. Division of Labour

These processes complement each other in helping any system remain secure from intruders working from the inside.

The Two-Man Rule

Let's start with the most obvious one, the Two-Man rule. An important part to IT and Infrastructure security is to make sure that all the behavior inside the system is identifiable and desired. By this implying that whatever action is taken inside the system is trusted.

When showing an example of this, my favorite way of explaining is the Git system of Forking and Pulling. In Git, everyone with access to the repository (The Infrastructure in this case) can make a copy. Then, people with access can request to pull their code into the repository. However, for this to happen, the pulled code must be analyzed, marked as compatible, and then authorized by someone else.

The same could be said and done for a secure Infrastructure. All management personnel can change the code, but for the changes to go into production, they must be approved by one or more staff.

The Accountability Rule

Another common problem with certain types of Systems and Networks is that there is one management account, who's password is known by all members with access. The first problem with accountability is raised here. Many companies, when in situations of rogue members making unauthorized changes in the server, rely on primitive methods such as checking the machine IP address, to locate who might have published changes to the system. This can be simply fixed by ensuring everyone has their own account, and making them aware that their changes are logged.

As mentioned in the last paragraph, logging is the second problem. The issue of trust rises to the surface again in this case. As the member is trusted to make certain changes to the system, the system is in most cases not properly logging the user's actions.

This situation is the perfect point to implement action accountability. The management user needs to be aware that not only are his/her actions tracked at all times while modifying the infrastructure, but they will also have contract-bound responsibilities and penalties for deliberate actions.

Division of Labour

This is another overlooked concept in most IT Infrastructure managerial positions. IT Teams have the tendency to divide their tasks, however, it is not uncommon for most users to have access to perform any task.

The best way to prevent this is to have specific system management tasks assigned to only two individuals (to prevent cases where one individual is not available). While other users can still verify and approve changes, using the Two-Man rule, only a handful of users can actually start those changes in the first place.

Personal Suggestion

A personal favorite way of implementing system-wide security, specially in large business environments is having 3 server sets. Alpha, Beta and Production, the first two being a clone of the latter. Anyone can move changes to Alpha, we use this system for testing how it would react in Production. Beta is for changes that have been tested and are ready to be deployed. To reach this stage, several members (~5) of the IT department must approve the change. When at this stage, the IT department also documents the changes, and sends them to Management and as a Memo to IT. To reach Production, 3 high-profile management members must approve the change, using their own accounts, which cannot be accessed by the IT department.

Last Note

As you may have noticed, this is not an easy process. Implementing many of these ideas will slow down production. This is one of the quintessential questions of Security. The more a system is secure, the more difficult it becomes to change and modify. To make your business productive, you must balance Security and Trust.

Answer 6

By ensuring that commands that would negatively affect infratructure in such a way it cannot be accessed remotely, can only be executed locally.

This can be achieved in a multiple ways. For example disallow shutdown command. Another way is to have a watchdog (hardware device that detects change in availablility) that will restart said infrastructure or run a recovery procedure if the infrastructure becomes unreachable.

A third way is to ensure remote access, by for example using KVM-Over-IP-solutions, and then tying these resources to controls that can only be manipulated on-site. Thus if infrastructure is brought down, it can easily be restored remotely.

Of course, its important to have backups of configuration files, important systems and such. Since configuration files are seldom changed, I would say a backup of config files would be good to do after each configuration change that is decided to be committed.

In case there is a need to disconnect infrastructure due to critical security events, a emergency system can be used, that will disconnect infrastructure in a way that its easily recoverable by management, without requiring a onsite visit.

---

The problem here was really not that rogue employee. What if that employee did just the same thing, but as a mistake. Lets say his intention was to repair a malfunctioning network device, not realizing that the equipment will be brought offline after a config erase, and does that particular code & command mentioned in the question, with the intention of recreating the config file after it have been erased, but after having executed the command, realizing "oops, can't connect to the device anymore".

And thus causes the same type of damage? Trust me, I have done the same mistake, mostly with my own systems, which have then required me to visit the physical location of the equipment. (but in that case the things were not critical, but if the systems are critical, you ought to do something to it)

Thats why it must exist safeties, so its impossible to cause that type of damage remotely, intentional or not.

Answer 7

Lots of good answers here, but one that seems to be missing is to embrace the Principle of Least Privilege (PoLP). If there's no legitimate use case for erasing the router config files, don't give anybody access to do that. If there is a legitimate use case but it's not relevant for day-to-day operations, require (and audit) an approval process to obtain that privilege (this is, in effect, a variation on the two-man rule that only applies to especially sensitive operations).

There's also backups and fail-safes. To take the original example, if a router's configuration is wiped, it should revert to an unremovable failsafe configuration (and raise an alarm). This should also happen if it fails an internal health check. Alternatively, if you have health checks, you could have the system revert to a "last known good" configuration, essentially restoring itself from a backup if anything goes wrong. Write access to these failsafe/backups/health checks should be under extremely tight security - nobody should have day-to-day privileges to them, or be able to get such privileges easily - such that even the most highly-trusted insider cannot unilaterally bypass or overwrite them.

Obviously, all of these solutions will have costs. There's almost always a tradeoff between security and expending resources (usually described as "convenience", but the resources can also be time and/or money). Really good PoLP means nobody gets genuine root (god-level) access to anything, for example, which slows things down for people who can probably be trusted with that much access (you can't ever truly know). Failsafe code is harder to write than code that just trusts whatever commands its fed from a "trustworthy" source, even if that command is HCF. Paranoia has its price... but that price may well be less than what it costs if you lose 90% of your network connectivity.

Answer 8

One single employee should never have the power to perform such widespread damage. Administrative actions like that should always require authentication by two or more separate administrators, where the authentication system can only be disabled by the top-level administrators.

In the event that it has already happened, the employer would have every right to fire the employee.

Answer 9

In every company what I've seen, there were very strict internal security rules as well. We couldn't see anything from others projects, only our actual tasks.

If we moved between projects/departments, our permissions were always precisely finetuned.

Only a narrow set of the sysadmins had access to large parts of the infrastructure, they all worked at least 5-10 years there. Probably nobody had access to the whole network.

Connecting anything to the company network without explicit permission (it was hopeless even to ask for that) was always strictly forbidden. Once I attached my smartphone to the usb port (only to load it), in 5 minutes I was asked by my coworker what I am doing.

Our computers/laptops were given to us pre-installed with the company certs. After leaving the company, we gave them back.

There were regular security checks, these were done by an external company (-> nobody knew, what and how they do). Also our software what we produced, were examined by them.

What we've done on the company network, were probably logged and backed up until the ethernity.

If we had done any doubtful, we hadn't know, how will be logs stored and examined. After leaving the company, our computers were also security checked by the external company, and probably backed up on the sector-level, for the case of a later appearing "problem", to serve as an evidence.