4

My team has had some challenges with our current internal PKI management tools and we are looking for other solutions. One of my team members forwarded me a link to a page about Lemur, an open source framework from Netflix.

It's a pretty well written article and most of what is discussed seems relevant to struggles I've seen and experienced but there's one thing that I'm confused about.

The following diagram is meant to show "a typical procurement process".

enter image description here

What I would typically do is generate a new key-pair on the host using something like the Java keytool. This generates the key and stores it in the keystore. Then a CSR is created and signed by the CA. The signed cert is then imported back into the store. I think this diagram can be interpreted to describe this process but it doesn't clearly show that the private key never leaves the 'deployment target'.

The proposed new procurement process looks like this:

enter image description here

Now, it would seem from the diagram, that the private key is generated on a separate server and then delivered (presumably via secure means) to the 'deployment target'.

The ability to generate a CSR and get the public key signed without needing to move a private key around seems pretty slick to me. I thought it was designed this way for a reason. While I can see the advantages from a convenience perspective, isn't this approach of pushing private keys inherently adding more risk of private key exposure?

The text of the article says that it can support certificates generated elsewhere which but implies that this is to help transition to using Lemur to generate them. I'm not convinced that is a step forward. Am I missing something?

Improve this question
8
  • The difference is Lemur does the CSR generation and sending the keys to the deployment target bit for the developer and handles the CSR approval bit for the Security Engineer thereby removing unnecessary manual steps in the process, especially as the developer may not get all the extensions correct on the initial CSR. In either case, the deployment machine needs the private key to perform key exchange and sign messages so one needs a way to put it on there securely anyway. In both cases in diagram that is only an intermediary, I guess Lemur has secret key storage built in.
    – ewanm89
    Nov 28, 2017 at 18:05
  • @ewanm89 "so one needs a way to put it on there securely anyway" I'm not sure I follow. Using the standard approach, the key is generated on the deployment server and stored there securely. There's no need to 'put' it there, if 'put' means deliver it over the network.
    – JimmyJames
    Nov 28, 2017 at 19:06
  • 1
    Except Netflix have a whole load of load balancers, applications servers, database servers, storage servers all needing certs then a whole separate development network. At the scale Netflix operates at, copying the private key from one machine to another will have to happen at some point. As the answer pointed out this is not for individual machines and this is for when one has a secure network to operate with.
    – ewanm89
    Nov 28, 2017 at 19:16
  • "Netflix have a whole load of load balancers, applications servers, database servers, storage servers all needing certs" So does my employer. The scale is different, to be sure but I don't see why the scale must require that "copying the private key from one machine to another will have to happen at some point".
    – JimmyJames
    Nov 28, 2017 at 19:32
  • @JimmyJames So you have multiple instances of the webserver host all using the same TLS certificate, but you have never copied the private key? How exactly?
    – Mike Ounsworth
    Nov 28, 2017 at 19:38

2 Answers 2

Reset to default
2

Yup, that's how I would read that too.

I assume that for a large cloud-based company like Netflix, the Lemur server would be part of their virtualized prod environment cluster and therefore held to the same level of network security as everything else in the prod environment. For example, if the deployment target is actually a cluster of VM hosts behind a load-balancer (or the load-balancer itself is terminating the TLS connections) then some amount of private key copying will be happening anyways and they'll have infrastructure to handle that properly. Adding a Lemur server to the cluster doesn't open any additional attack surface beyond what's already inherent to a large cloud deployment.

For smaller deployments where your prod server is a single machine, then I think your concerns are justified and it's worth asking some questions about additional copies of the private key floating around.

Improve this answer
20
  • "therefore held to the same level of network security as everything else in the prod environment": I guess I was reading from a perspective that perimeter security is bunk and that we need to assume we have internal threats. This approach would appear to create a situation where the comprise of a single server would compromise every private key used. I'm not sure if that qualifies as increasing the attack surface but it's seems undesirable.
    – JimmyJames
    Nov 28, 2017 at 19:24
  • I probably should ask the team directly and/or look into the details of the design but why not build the solution upon the existing CSR approach. If the deployment target can accept a key-pair from lemur, surely it can accept a 'generate a new pair and return a CSR' request followed by a 'here's a signed certificate' request.
    – JimmyJames
    Nov 28, 2017 at 19:27
  • "This approach would appear to create a situation where the comprise of a single server would compromise every private key used." Sure, but presumably a compromised admin account already has access to the key files on all servers. That seems like a very similar attack surface. On the other hand, if you're separating your prod servers into different admin groups so that no single admin has access to all of them, then you probably want separate instances of Lemur to match.
    – Mike Ounsworth
    Nov 28, 2017 at 19:35
  • OK, I see where you are going but it's not necessary to have the admin account credentials to be able to compromise the application and/or server. For example, a remote code execution exploit might allow for passively capturing private keys and pushing them to an accessible location.
    – JimmyJames
    Nov 28, 2017 at 20:37
  • @JimmyJames how you do you a remote code execution on the Lemur server with to publicly-accessible ports?
    – Mike Ounsworth
    Nov 28, 2017 at 20:46
1

Generally in the past Netflix has communicated that they deploy baked AMIs (VM images) of their code, which are stripped down to exactly what they need. This usually also means they are using an autoscaling group in Amazon that get spun up as they require. This is changing with containers but the same general principal holds. 1

This usually has one major implication for secrets like private keys. You aren't generating them on the host that will run your service & you do not bake the secrets into the application artifact. You pull them down at runtime or they are loaded onto a load balancer.

Second bit that I think Lemur has in mind and another product called Vault from HashiCorp pushes heavily is that you centralize where you generate/store secrets so you have a limited spread of where they are. It also means that you aren't baking extra stuff for cert generation into your production images. It does mean you probably have a few juicy targets, but the gist I've been seeing from a lot of these players is that it is easier to centrally protect a few things segregated by blast radius than to poorly protect hundreds or thousands of things. Usually this goes had in hand with shorter lived secrets so if/when things get compromised you just change it out and you have practice in doing it so the damage is limited.

Improve this answer
2
  • "centralize where you generate/store secrets so you have a limited spread of where they are" at a high level, I think that makes sense but in order to be used, these need to be distributed to the hosts that use them. It would seem to me that you are adding a juicy target and but not really limiting where they are. You also add a distribution step which adds more risk. I'm over-simplifying this though. If you are more concerned with a compromised host being used to generate a key that is then signed, I can see how centralizing this fits into the model you describe.
    – JimmyJames
    Dec 19, 2017 at 20:10
  • The other half I failed to mention of the centralization step for HashiCorp at least is that you do not have long lived certificates, they are constantly rotating so that even if a certificate or private key get compromised they have a very limited lifetime. We're talking days or a week at max.
    – sidewinder12s
    Dec 22, 2017 at 0:31

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .