This may sound like a stupid question but seriously how are private keys kept private?

If you're someone like Google you have some huge number of servers to which the public can establish secure connections.

The *.google.com private key is required in establishing every one of those connections.

Servers are presumably reaching the end of their life all the time and being disposed of (leading to the potential risk of leakage at that point of anything on them).

Modern data centers require very small numbers of people yet companies like Google are so huge that there's still a significant number of people with physical access to the servers.

These servers sit behind firewalls and run highly secured OSes but both routers and OSes may suffer zero-day exploits.

So it's fairly obvious that private keys can't be stored on a per server basis or on OS images retrieved on boot up (even if encrypted) or on per server smart cards.

Presumably companies don't discuss this too openly but are there any good overviews of how things work?

Is there some hardened bunker somewhere where just the initial asymmetric key element of establishing every connection is handled?

Presumably actually you need such bunkers per data center - I guess you don't want to lose an entire data center due to a losing your link to a remote key handling site.

Even duplicating the key across data centers sounds highly risky considering the value of a key like that for *.google.com.

Am I missing something that makes key handling like this not quite so hard as it sounds to someone trying to think it through from scratch?

  • 1
    To address one single point: in large organizations there are usually formal procedures to follow when servers are disposed of. At a minimum this would include wiping the contents of the disk drives, usually with multiple passes. If the drives in question contained particular sensitive data, they would probably be physically destroyed instead, e.g., melted. – Harry Johnston Apr 10 '15 at 5:06
up vote 82 down vote accepted

Firstly, often encryption is terminated at the perimeter by infrastructure which is dedicated to offloading SSL decryption. It makes it much easier to manage when you only have maintain a high degree of key security for a small (proportionally) group of servers which are dedicated to the role. The rest of your regular application servers can operate like normal without worrying about handling these keys.

Secondly, their keys would almost certainly be stored via a Hardware Security Module (HSM). These are dedicated hardware devices with a processor designed for maintaining security and being efficient at performing cryptography.

Finally, Google has its own intermediate CA certificate which it can use to sign its own leaf certificates. This allows them to use certificates with a far shorter expiry than normal, which somewhat reduces the risk of a key being compromised. The actual CA key can be kept locked away in an air gapped bunker and only accessed when it needs to sign a short term leaf certificate.

Remembering also that a CA doesn't need the leaf private key to sign a certificate, they could generate a private key which never leaves a HSM onsite at a remote data centre, then send the public key/CSR to the CA to be signed. They might even generate a unique private key for every single HSM device, that way a key never has to leave the device which generated it.

For example here's a print screen of a current Google certificate:

enter image description here

You'll notice that the certificate is only valid for 3 months, which is significantly shorter than most.


Edit: Google actually describes their policies in detail here.

In the Google PKI, a Subscriber is an individual or an organization capable of using, and authorized to use, the Private Key that corresponds to the Public Key listed in a Certificate

...

The Google Internet Authority servers are located inside of a locked cabinet or cage area in a locked server room. Access to the server room is controlled by badge readers. The private keys for the Google Internet Authority are stored in hardware security modules that are FIPS 140­-2 Level 2 that are physically tamper­-evident and tamper-­resistant.

...

Subscriber Key Pairs are generated (i) by the Subscriber by software supplied by their device/operating system, or (ii) by an authorized member of Google’s Information Security Team.

...

Subscribers provide their public key to Google for certification through a PKCS#10 Certificate Signing Request. The preferred transfer method for sending this information is HTTP over Secure Sockets Layer (SSL).

Basically that's exactly how a regular CA works, except it's all conducted internally within Google.

The short answer is "They are very very careful 'where' the private keys are."

I think you may be misunderstanding or ignoring a basic part of how these kinds of keys work - when you sign something, it's a one-way mathematical operation that "proves" it was signed by one particular entity. (Technically, someone with access to said key, and knowledge of any passwords/keys used to initially encrypt it itself.)

The servers you and I connect to don't store the private keys, they have "things" (usually certs in this case) that, being signed, could only be created by someone with the private key. In the same way that receiving a PGP-signed email doesn't let you then pretend to be the sender, a decommissioned webserver isn't in danger of (directly) leaking any keys.

This is all assuming perfect implementation of all stages of the process with no exploitable flaws in the code as implemented or the underlying mathematical algorithms, of course. Or someone with a big wrench and access to the relevant people. (https://xkcd.com/538/)

  • When one establishes a TLS connection with e.g. amazon.com aren't we generally using RSA to exchange an AES key? My understanding is that the client typically encrypts a random value of its choosing with the public key of the server. So won't the server (or something in front of it as covered by thexacre) need a copy of the corresponding private key? – George Hawkins Apr 15 '15 at 15:01

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.