2

(This question is based on a comment I made on How Vault agent solves Secret Zero challenge in Kubernetes?)

We are planning to run some of our software in a Kubernetes cluster running on bare metal (i.e. not using a cloud provider). We need to provide some secret data to our microservices (e.g. API keys, private keys), and want to ensure these secrets are stored and retrieved as securely as possible.

The Kubernetes Secrets documentation provides various recommendations:

  • Enable encryption at rest

    • The documentation for this states:

      Caution: Storing the raw encryption key in the EncryptionConfig only moderately improves your security posture, compared to no encryption. Please use kms provider for additional security.

      ...

      Encrypting Secrets with a locally managed key [i.e. not kms] protects against an etcd compromise, but it fails to protect against a host compromise. Since the encryption keys are stored on the host in the EncryptionConfiguration YAML file, a skilled attacker can access that file and extract the encryption keys.

  • Use an external secret store provider, e.g. Vault provider

    • This seems to be an alternative to using native Kubernetes secrets, and outsources secret management to an external component.

My thought process is:

  1. Using the kms provider seems to require us to use a Cloud provider - which isn't an option - or develop our own - and I assume "rolling our own" isn't recommended for this sort of thing. So we have to use an external secret store provider, of which Vault is the only supported provider.

  2. Vault requires the Pod to authenticate using its service account, which boils down to a JWT minted using a private key stored on a control plane node.

  3. This private key is no more protected than a static secret defined in an EncryptionConfig file, assuming our control plane is separate from our worker nodes.

  4. So, my question:

    What do we gain from using an external secret store, over using a static secret in an EncryptionConfig file?

    Is it just security theatre? Or is there a flaw in the reasoning above?

    Alternatively, is there a better way to manage Kubernetes secrets when running on bare metal?

1 Answer 1

0

We tried using bitnami-labs SealedSecret and quite happy with overall ease of use and also quite easy to setup Github: https://github.com/bitnami-labs/sealed-secrets

It's like extension to Kubernetes secrets. With default Kubernetes secrets, they are simply base64 encoded. But with Bitnami SealedSecret controller you have a base64-encoded asymmetrically encrypted Secrets.

The one way one-way encrypted Secrets (aka sealed secrets) you can store as part of your Helm/Infra-as-code and don't need external secret store. More details you can find on the Github documentation.

2
  • This seems to be just an alternative way of creating Secrets, that allows you to securely store the values in an IAC git repo - is that right?
    – PMah
    Commented Apr 27, 2023 at 9:11
  • yes that's what it is (encrypted secrets instead of kubernetes default base64 encoded) Commented Apr 28, 2023 at 10:56

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .