How should I set up emergency access to business-critical secrets in case I am "hit by a bus"?

Question

I work as the primary developer and IT administrator for a small business. I want to ensure that business can continue even if I suddenly become unavailable for some reason. Much of what I do requires access to a number of servers, (through key-based ssh), cloud services, and other secure infrastructure of applications. Some of these services use MFA, either using dedicated MFA apps (like Amazon) or SMS.

How do I ensure that my "hit by a bus" plan and documentation is complete and comprehensive, but that this documentation is not itself a security risk?

The documentation will be hosted on a shared file server behind our VPN, but that can also be accessed using a third party web frontend that puts a "DropBox"-like interface on top of the base file server (i.e. authentication, desktop syncing, file sharing, etc). The files are in a location where only I, and other file server administrators can see them.

How should I manage the "secrets" (passwords, private keys, MFA access) in this documentation to ensure it remains comprehensive without compromising security?

Answer 1

My advice would be to remove the secrets from the drop-box and store them elsewhere. Your instructions have to be easily human readable by anyone, but they can include instructions on how to get access to the properly secured part of the data. That lets you separate the accessibility side of things from the security side.

Once you can think about security on its own, you can start to ask the real question of how much do you need to protect these keys? This is a business logic question, so consult your management. You might:

• Have a password to a file that everyone knows
• Have a file set up with multiple passwords so that each individual maintains their own copy.
• Have a file locked by a M-out-of-N algorithm (the digital equivalent of requiring two keys to unlock a safe).
• Have a M-out-of-N algorithm with one 'master' password required regardless of which group of individuals unlocks the file, and that one master is physically kept in a tamper-evident safe that you check every now and then.

Use creativity here. Whatever you do, the decoupling of "instructions" with "sensitive information" frees you to properly safeguard the information, and then provide instructions on how to get that data later.

Your business logic decisions will also include uptime questions. If something goes wrong in your life, how long does another admin have to take over your work before business is affected? Consider how well replicated you want these instructions and sensitive information to be in case of server glitches. When I was administering a server and needed to store instructions on how to restore it from backup, I used the server's own wiki to store that information for easy viewing, but obviously that wouldn't be so useful in a glitch scenario, so I also had a copy on the dev VM of that machine, saved off copies of it on 3 separate PCs and a printout. I couldn't guarantee the printout would stay up to date, but I made sure that I could do my best.

This also points out something which is not always part of a hit by the bus plan: graceful degradation. Not all hit-by-the-bus scenarios involve getting hit by a bus. Some involve you just being inaccessible at an unfortunate time. Some involve you leaving the company, but being available for a question or two. Others... do not. Consider layering the plan. Small mishaps may be very well protected, while greater mishaps may still result in business loss while everyone gets things together, but nothing permanent. To use my backup restoration plan as an example, the printed version was almost guaranteed not to be fully up to date. But if lightning wiped out every computer for a city block, it was still more helpful than nothing. On the other hand, if the server just thew a harddrive, and I had to restore from backup, the version I kept sync'd on the dev box was almost certainly up to date.

Example of this failing: I was a user on a network managed by KERBEROS by an admin that was distrustful of others and did not have a hit-by-a-bus plan. When he... left, we had a hacking party to try to break his server. In the end, our best impromptu hit-by-a-bus plan was to wipe the machines (every one of them) and start from scratch. Note that, while this wasn't the best plan (in fact, I think its the worst plan?), the business kept moving. We just got stagnated for about two days and had a bunch of grumpy customers. In the words of Frank Herbert's Dune, "The spice must flow." Even in the worst case (which may involve a curious incident involving your server's harddrive flung out of the bus and hitting you on the head, destroying all record of the hit-by-a-bus plan), business does have a way to keep moving... but I approve of trying to raise the bar a wee bit more than that!

Answer 2

Get a USB device. Put all secrets on the USB, preferably in a KeePass file. In the documentation, tell the new person where the USB is and how to unlock it, but put the device in a secure physical location like the owner's office, the company safe, a secure deposit box, etc. Somewhere out of the reach of the public, and away from the prying eyes of other employees.

The advantages of this plan are:

• It's not on the internet or internal network
• You can be assured the new employee has at least managed to convince the person in charge of the storage location that they are authentic (and most likely will be flanked by high-level employees to even get near your storage location).
• If someone tries to get at the flashdrive before you, er, get hit by the proverbial bus, that someone should probably think to themselves, "Hmm, Andrew is still alive. Why is someone touching this?"
• If someone finds your documentation, the amount of damage they could do is limited (especially if they managed to break in over the internet -- very few people are going to road trip for your information).

To find the goodies, they need 1) your documentation, 2) physical access, 3) you to be 'pining for the fjords'. The USB is also a neat little package for the next person -- plug and play! Or panic, depending on how important you are to the company.

Answer 3

Create 'emergency use only' accounts

For most systems, you will have some kind of privileged accounts that are used in their everyday administration, those may or may not be personalized depending on your organization policy.

As for any credentials, you may lose access to them for various reasons - either by the person knowing them getting hit by a bus, or by losing the intended secure storage of that credential (e.g. losing a computer in a fire) or by somehow managing to change a password to something they don't know, etc.

What may be useful is to create separate accounts for key systems that have full access, but are not intended for everyday use. This implies:

1. At the moment, noone is unable to access them - no person knows the required passwords.
2. If someone accesses them, everyone should be informed (as it's not supposed to happen in normal circumstances). For example, automated scripts that email multiple key users upon login, all the actions from those accounts get logged and your everyday monitoring would light up if something is run from those accounts.
3. When required, people can get access to these accounts - a simple way is to generate a long password or key, print it out, put it in a sealed and signed envelope, and lock it in a safe.

Put your back-up procedures and credentials there

When you maintain your procedures, hit-by-a-bus plan and lists of secondary credentials in whatever way that's convenient for you - just make sure that these documents or password databases are also accessible from the emergency use account.

Answer 4

Maybe a better solution to your situation is to design the system such that even if you are hit by a bus, and your credentials are lost forever, nothing bad happens.

Perhaps it's best to think of the problem as two problems, authentication and authorization.

Authentication establishes that you are who you claim to be. Some system can know that a request it really came from you because only someone with your credentials could have made the request, and only you know the credentials.

Authorization establishes that you are allowed to do a certain thing. A request can be authentic but not authorized.

If at least two people are authorized to do a thing, then it doesn't matter if one of them is hit by a bus. Alice may be killed by a bus, and knowledge of her credentials lost forever. But Bob knows his own credentials, and he's authorized to do anything Alice could do.

If the system is designed in such a way that anyone could be literally hit by a bus and killed, it also means it's possible to revoke one person's credentials. Former employees, especially those that leave on bad terms, are a security liability. You can't force a former employee to forget a password, so to be safe you should change all your shared passwords whenever an employee leaves. In practice this is too much of a pain and it isn't done. Not sharing passwords in the first place solves the problem.

Never sharing passwords also preserves accountability. You can't run a business with no administrators, but you'd like to have the ability to know if a particular admin abuses his powers. Ultimately the threat of termination or litigation dissuades any admin from abuse. If passwords are shared then "must have been one of the other admins with the password" is a plausible defense.

Sometimes you run into situations where it seems impossible to avoid sharing a password. But there's usually a solution, even if it's not immediately obvious.

Do you need to share root passwords? No, you can not have a root password at all, and instead use sudo.

What about AWS root credentials? You can use IAM instead.

Maybe you have an especially brain-dead web application you can't avoid? You may not be able to fix this one but you can cover it up: use a password manager that allows sharing credentials among multiple users. (I know LastPass can do this). While you are still sharing the password, you at least have an automated means for changing it and distributing knowledge of the new password. Change the password at least whenever an employee leaves, but preferably more frequently.

Answer 5

Most of the answers here relate to the handling of credentials, probably due to this explicit question in the OP:

What's the best way to manage the "secrets" (passwords, private keys, MFA access) in this documentation to ensure it remains comprehensive without compromising security?

However, the title asks a different question, which I don't see addressed:

Developing a secure hit by a bus plan

The answer the that question is automation.

In devops, I find that most of the server side of the position falls into two categories:

• Fix the infrastructure: Here is usually nothing to automate: every problem is different. In those cases familiarity with the infrastructure (servers, network, how the devs interact with the Git repos) is key.

• Maintain the infrastructure: Create new VM instances, set up Apache, enable dev access to resources, etc. This is the place where automation is most visible.

Those the role of automation in the latter is obvious, the key to successfully fixing problems in the former is automation in the latter. The automated Python and Bash scripts serve as living documentation of what is expected of the servers and network: when the workflow changes it is the scripts that change. Git records changes that may be applicable to older servers, and git commit messages are explicit.

Answer 6

This isn't another idea on how to do it per se, rather a point to flag up something that has not yet been discussed but is very important. I'm proceeding from a standpoint that these secrets are really business critical.

Testing.

Some disaster recovery scenarios have been posited. Some are simple, some are more complex. The more the complexity, the more reason to test. This issue comes from how do you test some one being 'removed' from the scene in a business critical environment? Putting a 20mm round through your production server during Christmas trading, with your primary IT guy incommunicado will be difficult to get though the board of your toy factory. This is the dilemma facing the board. If it's important, you need to test it. Is it too important to test though?

Simple things will catch you out. Some one has mentioned putting data into safes. Great. A lot of people keep their safes in the basement where it's secure and away from people. It's also the place that initially fills with water following a minor fire. Another; your dodgy payroll clerk has been doing really dodgy stuff with your payroll. The police come in and confiscate all your servers for the ensuing investigation. It takes a year to get them back. Don't roll your eyes - it happens.

Business continuity is a fairly established art, so just do what NASA, Google, Barclays, the army and nuclear power stations do. Make redundancy. I'm sorry to say it Andrew, but you're the principle risk to the company in this scenario. The board need to be made aware of this, and you and at least some of your kit needs to be made redundant (in the duplication sense).

In the meantime, get your ops director to get some critical person insurance and /or business continuity insurance.

Answer 7

I'm in a very similar situation as the sole IT admin for a mid-size company with a lot of scattered, loosely related custom software on different platforms. Worse than that, I wrote all the software for the company over the last ten years, from their POS system to online reservations, a homebrew CMS, employee training software, etc. At this point I'm the only person who understands or has ever even seen the source for those things. As the company has grown, I've been asked more than once not to get hit by a bus. I'll tell you what our solution to this has been.

1. Understand that there will be disruption. Manage expectations. If the company doesn't want to hire a redundant person, presumably a well-paid one, to learn everything you know, then they need to expect there will be a very steep learning curve for anyone who needs to come in during an emergency and try to fill your shoes. This is true no matter how great your documentation is.

2. Make sure the service bills are going to the company, not to you.

3. Continuity of business document. At a point I was asked to write a doc that lays out in plain English, that the CEO of the company can read, exactly what servers we have, what is on each server, what each piece of software does, and the passwords for those accounts. Within that are notes and tips for anyone who came along and needed to keep it running. However per (1) it is understood that it would take a book to explain in detail the functions, known issues and limitations, and structure of all the software. So this document contains the bare essentials, with the hope that once someone got into the code they will start looking at cron tasks and reading comments and get a handle on how it works.

I update the continuity document when necessary, PGP encrypt it so it can only be opened by the CEO of the company and myself, and upload it on their webserver to an address he knows. The CEO is responsible for keeping his PGP key safe. That's all there is to it.

And listen - if they're not even gonna let you relax after you're dead, it's time to ask for a raise.

Answer 8

Your solution, in order to be comprehensive, must be a combination of procedural and technical controls. The best way to understand it is to see technological controls as providing the capability to enhance your procedural security. They make some controls possible, and others easier to execute.

With this in mind, it will largely depend on how you're using your PKI. If you're in a smaller organization, budget constraints may limit what you can do, but if possible, I'd recommend a Hardware Security Module (HSM) to protect your private keys--at the very least, the private keys of your internal CA, if you have one. Vendors in this space include Thales and SafeNet.

As for passwords, there are a number of password management solutions designed to service organizations. The most important thing to look for is either one of two things: A solution which offers the ability to entitle a user with the access privileges necessary to do their job within a pre-defined time window without having to give said user the actual password to the system/application, AND/OR a system designed to support split-secret knowledge (where no one individual has the complete password).

Some security vendors offer a means to create additional role separation in what an AD Domain Admin or DB admin can do (for example, they may be able to add users to groups, but not enable the ability to actually open/modify files that membership of that group normally entitles them to--that ability would be handled by yet another admin who likewise cannot add/remove users to groups). One such vendor is Vormetric--they do this with their data at rest encryption product.

While executive support is important to the success of any organization's security strategy, it becomes especially vital in the absence of a vendor solution which provides these capabilities--because that means you'll likely need to step up your security awareness programs to foster the kind of culture change that will support the necessary procedural controls most IT users aren't used to--where domain and enterprise admin privileges are strictly controlled, and entitled only after change management approval is granted, and only during a specific time window. IT admins must incorporate multi-person password + Physical controls.

Examples include using 2-4 people to generate halves of a 12-character password. One pair of personnel generates the password and enters it into the "New Password" field. The second pair re-enters the password components that the first pair created (to help ensure one of the first pair is not maliciously or unintentionally mis-typing the password each time, rendering the account inaccessible once the password is changed). Using a second pair of personnel also ensures that the passwords are legible as-written. Once the password is changed, the password fragments are then stored in separate opaque envelopes, sealed, labeled, and deposited in tamper-evident bags. Bags can then be dropped in a physical safe. Replicate the process for off-site backup storage, but incorporate processes to ensure password updates are replicated each time for each site. If tamper-evident storage is not enough, look for a GSA-approved double-combination safe (X-10 Kaba Mas, for exmaple), and incorporate the same split-knowledge procedure. For additional protection, store the safe in a location with a CCTV camera system monitoring the area, and incorporate a warning to all employees that anyone found in the area without prior written permission to enter may be subject to termination--again, this should come from the top.

It really just depends on your threat model and what appetite your organization has for security.