Secure private keys in unattended servers

Question

Suppose I have a server (e.g. an internet connected raspberry pi) that needs to autonomously access some third party service (e.g. stripe, aws, etc.). To use this third party service via their SDK, I need to provide it with a secret key in order to initialize the SDK. My question is, how do I store this secret key in the server securely?

I don't want to store the key in plaintext, but encrypting it seems like it wouldn't help. If it is encrypted, I then have to provide some sort of credentials that the server can use to unencrypt the keys. Then these credentials are on the server, and getting access to those lets you unencrypt the key anyways so nothing changes.

I feel like encrypting keys and then providing the decryption password is like putting valuables inside nested shoeboxes and saying "HA! They will never be able to get to the inner shoe box because they have to open the outer one first!"

What is the best practice here?

Answer 1

Let's start with the constraints:

1. You need some kind of shared secret to send to a service to authenticate, so you need to be able to access the plaintext.
2. This needs to be done in an automated fashion (unattended).
3. You'd like to protect the credentials as much as possible.

First, we have to acknowledge that it is impossible to completely protect them, and it may or may not be worth doing so. If someone can execute code on your server, they can do almost anything the legitimate service can -- including reading the keys and using the keys to make requests to remote services. Depending on how much inconvenience you're willing to suffer, however, you can make them harder to use.

First, let's start with the "best practices" that have little to no impact on the ability to use these.

1. Run each service on your server as a separate user. Ideally, add a MAC system like SELinux or AppArmor to further contain a compromise.
2. Store keys/secrets in files that are only readable by the proper service and lock down the permissions as much as possible. (0400 on Linux.)
3. Ensure that the keys provide as little access as necessary on the remote end. Uploading files to an S3 bucket? Create an Access Token with only that permission. (This way, if compromised, it limits the damage an attacker can do.)

If you need to go beyond this point, you can do a few things:

1. Store the secrets in a hardware module. (Particularly if asymmetric crypto is an option.) This won't prevent an attacker from using your credentials, but will prevent them from exfiltrating them.
2. Use a hardware module that requires human interaction. When the service starts, you'll need to enter a PIN, and then the service keeps the decrypted key in memory only. (Of course, a good attacker can probably get the key out of memory, but it raises the bar.)

Beyond this point, we get into the "silly for most applications": air gapping, interactive tokens requiring PINs for each signature, etc. The unattended nature will quickly go away.

Answer 2

If the server is compromised there is no way to do this securely because the attacker can do whatever he wants e.g intercept the password. Look at the approach of public keys in gnupg: they store the encryped secret on disk and decrypt using a user provided key into memory. This approach works if you can have user interaction. By my knowledge if there is no human interaction it is hard to secure this key.