How are private keys kept private?

Question

This may sound like a stupid question but seriously how are private keys kept private?

If you're someone like Google you have some huge number of servers to which the public can establish secure connections.

The *.google.com private key is required in establishing every one of those connections.

Servers are presumably reaching the end of their life all the time and being disposed of (leading to the potential risk of leakage at that point of anything on them).

Modern data centers require very small numbers of people yet companies like Google are so huge that there's still a significant number of people with physical access to the servers.

These servers sit behind firewalls and run highly secured OSes but both routers and OSes may suffer zero-day exploits.

So it's fairly obvious that private keys can't be stored on a per server basis or on OS images retrieved on boot up (even if encrypted) or on per server smart cards.

Presumably companies don't discuss this too openly but are there any good overviews of how things work?

Is there some hardened bunker somewhere where just the initial asymmetric key element of establishing every connection is handled?

Presumably actually you need such bunkers per data center - I guess you don't want to lose an entire data center due to a losing your link to a remote key handling site.

Even duplicating the key across data centers sounds highly risky considering the value of a key like that for *.google.com.

Am I missing something that makes key handling like this not quite so hard as it sounds to someone trying to think it through from scratch?

Answer 1

Firstly, often encryption is terminated at the perimeter by infrastructure which is dedicated to offloading SSL decryption. It makes it much easier to manage when you only have maintain a high degree of key security for a small (proportionally) group of servers which are dedicated to the role. The rest of your regular application servers can operate like normal without worrying about handling these keys.

Secondly, their keys would almost certainly be stored via a Hardware Security Module (HSM). These are dedicated hardware devices with a processor designed for maintaining security and being efficient at performing cryptography.

Finally, Google has its own intermediate CA certificate which it can use to sign its own leaf certificates. This allows them to use certificates with a far shorter expiry than normal, which somewhat reduces the risk of a key being compromised. The actual CA key can be kept locked away in an air gapped bunker and only accessed when it needs to sign a short term leaf certificate.

Remembering also that a CA doesn't need the leaf private key to sign a certificate, they could generate a private key which never leaves a HSM onsite at a remote data centre, then send the public key/CSR to the CA to be signed. They might even generate a unique private key for every single HSM device, that way a key never has to leave the device which generated it.

For example here's a print screen of a current Google certificate:

You'll notice that the certificate is only valid for 3 months, which is significantly shorter than most.

---

Edit: Google actually describes their policies in detail here.

In the Google PKI, a Subscriber is an individual or an organization capable of using, and authorized to use, the Private Key that corresponds to the Public Key listed in a Certificate

...

The Google Internet Authority servers are located inside of a locked cabinet or cage area in a locked server room. Access to the server room is controlled by badge readers. The private keys for the Google Internet Authority are stored in hardware security modules that are FIPS 140­-2 Level 2 that are physically tamper­-evident and tamper-­resistant.

...

Subscriber Key Pairs are generated (i) by the Subscriber by software supplied by their device/operating system, or (ii) by an authorized member of Google’s Information Security Team.

...

Subscribers provide their public key to Google for certification through a PKCS#10 Certificate Signing Request. The preferred transfer method for sending this information is HTTP over Secure Sockets Layer (SSL).

Basically that's exactly how a regular CA works, except it's all conducted internally within Google.