How do very big companies manage their most important passwords / keys?

Question

Third-party password managers such as 1password, etc. are useful for people, businesses, etc. to store passwords. But I bet Facebook, Google, Twitter and other super big tech companies don't use such third-party services for their internal passwords and have their own password managers for their most critical passwords.

How can a very big company manage some of the world's most sensitive passwords? (example: Gmail team root access password!)

Even with the most advanced password manager, you still have the problem of the master password.

Should this be shared among a few trusted people? Or kept by only 1 or 2 people (then what happens in the case of an accident?)

Are big companies known to use implementations of Shamir's Secret Sharing?

More generally, what are well-known methods that very big companies use to manage their most sensitive passwords? (i.e. passwords that, if lost, could generate tens of billions of $ of loss)

---

Note: I'm not speaking about the usual login passwords for each employee, but more about the company's most important passwords / encryption keys / private keys / root passwords, etc. i.e. passwords that if lost / compromised would have very severe consequences.

Examples (well I'm sure you can help me find better examples...):

• for a company like Facebook, how to keep the password of the "admin panel" where DNS records of www.facebook.com are set up?

• how can a company like Coinbase keep the private keys of their cold storage accounts? (worth billions of $ probably?) If multiple people have them, the probability of someone becoming crazy and running away with the keys is high. On the other hand, if only one person has them (e.g. CEO), then when he's no longer alive, the company (and its customers' accounts) is suddenly worth 0$.

Answer 1

In General

There's not really one answer to this question, and I wouldn't necessarily consider "large companies" to be a distinct thing with different approaches. Certainly, the particular companies you have named have their own way of doing things, but the people who would be best able to answer for them are employees of those companies.

As for large companies in general they vary too widely, even for "tech" companies. Some take a very top-down approach to security and tech usage, and some don't. Some care a lot about security and many don't. Some companies prefer building their own tools, while many prefer using 3rd party systems and not re-inventing the wheel every time.

In addition it can vary wildly inside a business. Different departments inside the organization may have their own methods, different teams inside departments may also do things differently, and then of course individual employees may frequently do their own thing. Again, it depends a lot on whether or not technology choices are top-down or more bottom-up.

An Example

The company I work for is a larger (~12,000 employee) international e-commerce company. We use Single Sign On (SSO) for most everything internal, but still provide employees with an account with an online password manager. This is intended to be used for all the other websites that employees have to sign up for in the course of their job which they cannot login to using our own SSO (AKA the rest of the internet). The company also purchases a license for a "personal" account with the password manager for every employee, mainly in hopes that it will encourage people not to store personal passwords in their company account.

In practice usage of this varies wildly from "department" to "department", with some having almost 100% adoption while others areas of the business use it very little at all and have their own preferred methods of storing important secrets.

Answer 2

Facebook authentication, when I left, focused heavily on multifactor authentication. Both Facebook and Google invested in purchasing Yubikeys, and Google went on to develop U2F which became FIDO. Server access was based on signed SSH certificate issuance. There was a "break glass" ssh key that was physically stored in a safe as well as a few "super bastion" hosts that could be used in case the site failed so badly that people started calling the police. IP ranges and DNS records, however, can be as simple as auto-applying configuration from a git repository. DNS changes at Facebook are frequent.

Almost all access is based on regular user identity, but usually with layers. A VPN connection that requires a certificate coupled with a signature challenge where the key is stored in the computer's secure enclave may be used such that you can only access a system with my credentials if a computer assigned to me is involved.

In smaller companies using AWS, I've implemented AWS root access control by putting the password and 2FA secret into Vault and used Vault's built-in SSS root key distribution for backups. Any access is audit logged each time because you need the rotating 2FA code (Vault won't let you read the secret back out, only the token value) unless you collaborate to acquire both the backups (managed by one team) and the necessary collusion of other employees.

Answer 3

I work for 1Password.

We do have a number of large companies using 1Password, but we do not talk about our customers without their express permission. Note also that we can't see what is being stored where, so we can't really infer from what we can see how they are handling some of the questions very good management questions you asked. However, we sometimes are part of discussions with the security teams of our customers about such things.

Some obscurity

One thing to note is that while security through obscurity is generally a bad thing, some of the details you mention are best kept quiet. If, for example, an organization is using Shamir Secret Sharing, then you may not wish to publicize who holds shares and how many shares are needed.

So I am not going to tell you who are the admins of our 1Password account nor how they manage their own master passwords and secret keys. Nor will I detail the aspects of our business continuity plans that deal with some of those people getting hit by a bus. These are well thought out and well protected, but I don't want to put a target on anyone's back. (Sure you can guess, and some of those guesses might even be correct.)

Recovery power splitting

One thing that you might want to do if you are an enterprise using 1Password is try to make sure that you limit the number of people who have both recovery powers within 1Password and control of email within the organization. I don't want to go into the gory details of all of the key management used for account recovery, other than to note that we at 1Password never have the keys to be able to do so, but if Alice is both the right sort of admin for a 1Password team that includes Bob and she can read Bob's email, then she has the power to take over Bob's 1Password account on that team. (Though not in a way that is invisible to Bob.) Note that this is documented in our security design document.

So some organizations may wish to limit the people who would have both powers. This is a bigger problem for smaller organizations than larger ones. In smaller ones, you will have smaller IT teams, and so people who may be expected to perform account recovery may also be the manager of organization email addresses. This, by the way, is one of the reasons that we offer free family accounts for members of a business account. The employer has no ability to perform any recovery or access to the data in an employee's family account.

Finer grained admin powers

Being a member of the Recovery Group for a team means that certain keys have been encrypted to your public key. There is a great deal of administrative tasks that don't require being a member of the Recovery Group. An enterprise can safely automate provisioning and deprovisioning users for example without ever being able to access or decrypt recovery group keys.

In general, we try to make it easy (or at least not too painful) for organizations to follow a least privilege policy for the powers that are involved in managing 1Password users.

Shamir secret sharing, HSMs, etc.

1Password does not (yet?) offer this technology. But I have reason to believe that some of our customers do this on their own for some master secrets. Likewise, I have reason to believe that some of our customers are using HSMs for decrypting some master secrets.

I believe that it is a good thing that they are doing so outside of 1Password tooling itself. We could do more to provide hooks to make such integration easier, but that key management should be through some other system.

I, too, would love to know more about what our customers are doing with this, and so I look forward to following answers. But at the same time, I believe that this is question of policies and practices where some obscurity is useful.

Answer 4

You mention private keys. For these, a well known method is to use hardware security modules (HSM). Like chip-based credit cards, they keep the key inside a box you cannot open, and you store the box in a safe location. Access to the signing feature of the box (without revealing the secret key, of course) may also be protected electronically, like your credit card requiring a PIN number. Boxes may be plugged in directly to a server if you need to use the key often, or they may be stored offline.

HSMs are usually only one part of a bigger infrastructure to protect keys while still being able to use them, but companies are not keen at showing in great details how they do. IANA, while not a very big company, is however very open about this. And it "owns" incredibly important keys. Their Root Key Signing Key ceremonies are video recorded and the videos are published online as part of their procedures to build trust with the public. HSMs are stored in a safe and only connected to fairly trusted devices (a read-only operating system on a computer which is also stored in a safe). Signing a key takes about 3 hours as many steps are needed to safely bring the data of a signing request, initially stored on a USB key, to the HSM, and then to get signed data back on the USB stick. Finally, the process requires the physical presence of several humans that should not trust each other.

Answer 5

@CaffeineAddiction Thanks. I'm specifically speaking about important keys like encryption keys, root passwords, etc. Who keeps them? the CEO/CTO and a few trusted employees?

It depends on who needs access, and the hierarchy in the company. Larger companies typically have multiple departments comprised of multiple teams. And not all staff in each department will require the same type of access.

There are multiple solutions for storing secrets and managing access to them. I will highlight one with which I am most familiar, HashiCorp Vault:

Secure, store and tightly control access to tokens, passwords, certificates, encryption keys for protecting secrets and other sensitive data using a UI, CLI, or HTTP API.

I have also personally used a combination of disk and file encryption techniques in the past, to secure access to these, e.g., dm-crypt, and gpg.

Answer 6

While technical solutions are great, the reality is that many companies don't use them. And often it's a matter of inertia.

I have worked in a variety of companies, from tiny 2-people startups to massive FTSE-100 multinational. What you'll find is that small, agile companies are usually way ahead of large incumbent multinational in terms of technological solutions.

The unfortunate reality is that many large corporations still use shared spreadsheets with passwords in them. Many still rely on people's memory. For example, in my present role of a mid-senior management in a large multinational, I have responsibility for and access to systems where, if abused, it could completely bring down a multi-billion pound company. Some of these use SSO and authenticate against the company LDAP. Yet, some rely on shared access, i.e. one login shared by multiple people. When I started in this role, I was told the username/password verbally and told to never write it down or share with anyone.

Ever since I joined, I have been pushing for vault and password management solutions for such tasks. Unfortunately, the brick wall I hit is our equivalent of CTO (official title is different, but irrelevant here), who is adamantly against any electronic password managers or vaults (his argument - "I don't trust them, don't bother bringing this up again"). And so we continue with spreadsheets for many of the passwords.

The specific solution I tried to push for was a local installation of a known open-source password manager (won't name it here). It allows for users to add passwords to it and share them with other users on the same installation. That way, there is no single password to remember. Shared passwords are stored in a nameless account and shared with other users that need them.

Answer 7

Excellent question!

Disclaimer: I've worked for large tech companies, and this answer is based on that. No company-specific or proprietary techniques are disclosed.

---

Authenticating people

I bet Facebook, Google, Twitter and other super big tech companies don't use such third-party services for their internal passwords

Actually, at least some do use third-party password managers - for employees and non-critical services. By the nature of the business, employees often need to interact with third party websites (employee information management, travel booking, employee credit cards, ...). Note these are individual credentials - they authenticate a person, not a resource or process

The largest of the services will support SSO (single sign-on) provided by the company. SSO is a lot more secure, but not all vendors will support it.

Another adopted practice in most large tech companies is the use of security keys for two factor authentication using U2F/FIDO or more recently WebAuthn/FIDO2

TOTP ("Google Authenticator") is common too, but more vulnerable to MITM attacks.

---

Authenticating resources and processes

How can a very big company manage some of the world's most sensitive passwords? (example: Gmail team root access password!)

There's no such thing as the "gmail team root password". Its existence would be a huge threat to the privacy of user data - and by extension, to the company.

There's a subtle difference with your last case here. We're not authenticating people - we're authenticating resources and processes.

There's usually no need or benefit to use a password for those cases, but they're still used for convenience, ease of implementation, or because there's no other alternative.

Let's walk through some scenarios inspired in real-life examples at large tech companies:

---

The 8 digit pin to the safe containing the cable that connects you to the data center

This safe might or might not actually hold a cable

Here we're talking about a shared resource that is authenticated using a shared credential. There's no (easy) way of making it more secure by making it support individual credentials, or isolated access.

Other examples:

• gate codes
• alarm disable codes
• emergency "break-glass"-for-key credentials

---

The best practice for this type of resource is:

• Secret sharing through a secure channel (such as a password manager)
• Secret access auditing (individuals are authenticated, and access to the secret is logged)
• Secret rotation - the secret is changed periodically

Of course, you will also find sub-par practices often!

• Secret sharing in a spreadsheet
• Secret scribbled down in some post-it
• Secret known by this one guy that people know

---

Scenario 2 - The software process that stores user email

Would you trust this random piece of code to store your most intimate secrets?

This scenario is more complex. Here we have a process that needs to identify itself to other processes (such as a database, or a web server).

---

The best practices can include:

• No usage of passwords (or shared secrets) at all.
• Usage of short-lived authentication tokens, common in protocols like Kerberos
• Usage of key management platforms like Hashicorp Vault or AWS Key Management Services, which facilitate secret generation, rotation and authentication.
• Usage of Hardware Security Modules, ensuring that access to a physical device must occur for the process to perform its function
• Usage of code review protocols. This serves a dual purpose of access control and auditing, to ensure that no single person can gain control of the system

Worst practices include:

• Hardcoding passwords / keys into source code

• Letting random people change the code with no approval at 4am

---

Scenario 3 - the worldwide admin group that can reboot all the servers

You know you have a problem when you need to explain to this many people not to press the red button

Here we have an action or task that can be performed by certain types of people.

The best practice is the usage of a proper Role-based access control (RBAC) system. These are often tailor-made, and vary from organization to organization. A primitive example is a UNIX group.

As in the "shared credential" example, auditing / logging is essential, and easier in this case, since the resource is electronic / software-based. Group membership can be freely given or taken, either by an administrator group (which is just another group!) or even by a quorum of members of the group itself

---

About the importance of processes vs tech

Are big companies known to use implementations of Shamir's Secret Sharing?

Kind of. Even in the realm of crypto-nerds conducting super-secure computation, you'll find that in practice human processes are often just as important as technical ones.

Famous examples:

• The IANA DNSSEC Root Signing Cerimony, which secures virtually all the internet
• The multiple codes held by different people needed in order to launch a nuclear attack

Answer 8

Risk Management, and Role Based Access Controls (RBAC)

OP is asking for technological solutions to what is, ultimately, a problem around Risk. Other answers provide a good sampling of the kinds of tools available (e.g. 1Password, Hashicorp Vault, etc...) but I'd like to address the underlying question of risks. That will determine which of the many options make the most sense. There is no one-size-fits-all answer, as different businesses have different threat models, and face different sources of risk.

OP basically asks about the risk of:

• Sole controller of secret goes rogue or is unavailable
• One of many controllers leak the secret or goes rogue

The usual measure of Risk is probability of loss x value of loss. For example, you have a 1% chance to lose 1 million dollars per year to $BAD_THING. If you can buy insurance for less than $10,000 against that loss, it's worthwhile to do so. You are pricing your risk, and ensuring against it. If the insurance is too expensive, and you choose to do nothing and hope you get lucky, you are accepting the risk. If you put in place policies and controls to stop $BAD_THING from happening, your are mitigating the risk.

Businesses evaluate the loss potential of losing (or having stolen) their crypto keys, put a price on that event, and then look for controls to evaluate costs.

"Many controllers" of the keys lets the business trade one risk for an other, which may be more palatable. Single controller gets hit by a bus, and the one and only key is denied to the business is a pretty big existential threat if everything depends on that one key/controller. So you give controller access to multiple people. Now you don't lose access to the key, but you increase the chance of "going rogue." How likely is that? Pick a number. 1% chance per employee with access per year? You can now start to model how many people should have access at any given time.

This leads you to RBAC. "Jane the CTO" should never have personal access to SuperSecret. However, she may be the only trusted member of the group SecretAdmins. That group has access to the SuperSecret, but membership in that group is dynamic, can be audited, and changed as needed.

This gives the business the ability to adjust risk appetite, and balance competing interests against different sources of risk.

The specific tech is almost irrelevant. The important part is that the design of secret control matches the risk profile.

Answer 9

The most sensitive keys should be generated and stored on Hardware Security Modules (HSMs) and never leave them. Security then becomes one of physical access to the HSM itself, plus some way to manage revoking the key if the device were stolen. The most obvious example for this being sufficient is managing the private keys to web server TLS certificates. If the HSM breaks, you just get a new certificate. If it gets stolen, you revoke the certificate.

For the case when losing the key would be a significant problem, the main idea is to store the key in one or more HSMs and only use it to encrypt other keys (you store the encrypted keys securely somewhere, but it is not the end of the world if the encrypted keys get stolen), the way TLS Root Certificates are used to sign intermediate certificates, and then those keys encrypt other keys, and those other keys are less valuable and easy to replace, and they are the ones that get used for something other than encrypting keys. With public key cryptography this is much, much easier than it used to be with only symmetric keys.

The master key is split and distributed, but how that is handled is particular to each company. You can use separate HSMs to generate and store separate keys and then combine them to form the master key simply by concatenation. Or you can XOR them together. In many cases, just splitting the key in two is considered sufficient. Any key that I have personal knowledge of other than TLS Certificate Authority root keys has only been split into 2 or 3 parts, either by cutting the key in pieces or by XORing the key with random numbers. I would hope people are using Shamir's Secret Sharing more widely now, but I do not know.

The key part can be stored on an HSM, a USB stick, or plain paper. It can be stored in vault or a secure cabinet in a secure facility (for example, the kind of building where banks process wire transfers is secure enough that a locked cabinet in a locked room in such a building could be considered secure enough to hold a key part if no other part of the key were in the same building). Or it can be stored in the CEO's wallet.

For a distributed key, the important thing is to make sure any theft of any part is detected so the key can be changed. Some systems have primary and secondary key already provisioned so the primary key can be revoked in an emergency without having to go through the process of provisioning or installing a new key first. So the CEO may keep their share in their wallet so that in case they are kidnapped, they will have no trouble handing it over rather than trying to endure whatever the kidnappers have in mind for applying pressure. Security is always a bunch of trade-offs.

Answer 10

Worth revisiting is the RSA hack from 2011. The fact that their private token keys (or seeds, which can be turned into keys with a reverse-engineered algorithm) were not held physically separate came as shock to many of us. At the time, I had more confidence that once I was logged into my RSA 2FA token secured corporate account, I was truly working in a secure tunnel.

Oh well, so much for that.

My employer apparently has similar misgivings, as they now forbid US employees to access from outside the US without major approvals from many levels up. It's now a severe offense to just take a company-owned encrypted device out of country, much less turn it on. (This has wreaked havoc on expats and immigrant employees who could return to visit family in country of origin for a few weeks.)

From what I see in the trade media, companies are taking more seriously the need for air (and steel safe) gapped, fully disconnected, physically secured storage of keys. One can hope measures such as those taken for the DNSSEC Root Signing Ceremony is an example. We're a long way from the days when Jon Postel could take over DNS just to make a point. (Or are we?)

Answer 11

But I bet Facebook, Google, Twitter and other super big tech companies don't use such third-party services for their internal passwords and have their own password managers for their most critical passwords.

Funny that you mention Twitter. Reportedly:

hackers breached employee Slack accounts and found credentials for the Twitter backend pinned inside a Slack channel.

If this is true, it means that rather crucial passwords that allow impersonating users are (were?) just pinned on a board where employees can access them.

I am aware of (not so large) companies storing all of the master passwords in a google sheet or a text file. The best practice that I have ever seen in real life is a shared KeePass file.

Answer 12

Most large customers I've worked with -- several with more than 20,000 unix hosts -- simply don't trust passwords on their own. Think large finance, energy, and other highly regulated industries.

They use a product like "Powertech BoKS" to control access by separating authentication from authorization. For example, it doesn't matter if you know the root password (authentication) if you don't have a rule (authorization) allowing you to use it. And yes, that includes at the console!

Integrating tokens and strong access control is probably the #1 method of securing these large environments. They do not simply rely on password knowledge to determine access.

For example: Access to a server (via SSH, telnet, ftp, x, etc) is only allowed for specific users... even if many users have accounts on a host, maybe some are allowed to use SSH, some allowed to use SCP, and others allowed to use FTP. On top of that, they are only allowed to come from identified known sources.

A sysadmin might be able to login to a host using his personal username password... but, to switch to the root account, he must use 2FA. And... even if he has a valid 2FA response, he must have permission to switch to root or the access is denied. The admin has no knowledge of the root password.

Many of these companies also further limit the use of the root account to specific time windows. That means that even if the sysadmin knows the root password, or has a valid 2FA response to become root, his access would be denied if he tried to gain such access outside of a change-control window.

The admin also must come from a known host or network... ie. the connection can't originate from the DMZ, or maybe its only allowed when coming from a specific jump-host.

These kinds of tight controls are key in securing these large organizations.

As for the actual root password... because we all know at some point, someone is going to need it... The root password(s) are usually kept encrypted in a vault. A sysadmin can "check-out" the root password in the event of an emergency. Keep in mind that the ability to check out that password is also tightly controlled. In most cases, the admin cannot check it out unless he has a rule granting him access to the vault system itself, and a rule allowing the checkout of the specific root password for a specific host. He would also need to use his 2FA to access the vault to begin with. In some more advanced implementations, he can't check out the password without tying the checkout to a valid change-control ticket. Once checked-out, the vault will automatically randomize a new password for the root account.

These concepts hold true for many large companies, but also apply to companies with as few as 25 unix hosts. It's not so much a factor of how big the company is or how many servers they might have, it's how sensitive do they consider their data. Even small companies can be held to regulatory requirements that they must meet.