Security and practicality of passing secrets to services running as non-privileged user

Question

I strongly dislike and mistrust the practice to store secrets in plain text files for unattended access by services.

This is increasingly less of an issue in modern deployments where secrets are provided by an external context (could secret managers, hardware components, virtualization hosts etc.), but in many on premise installations, especially where security is no concern until it becomes one, nothing much has changed over the decades.

My question is this: What would be the best way to keep secrets on a local system but out of reach from non-privileged user processes. I assume that once an attacker has root access, there is very little that can be done.

My ideal solution would be something like this:

• Secrets are kept by one process, which also controls the life cycle of service processes, most likely systemd on Linux
• Secrets are passed according to service specifications:
  • only to the process started as a result of activating the service.
  • only when the process is started (no long lived access to the secret, no access for other processes of the same UID; f.e. passing it through stdin where it can be read once).
• Combined with secrets managers and one time passwords, this should severely limit the risk of secrets leaks.

systemd-creds seems to aim at solving at least parts of the problem by removing secrets storage away from the file system, but it's not yet available in most stable versions of popular distros.

It also does not seem to address the issue that a UID can be more easily compromised than a single process and in most cases it's actually only the single process that actually requires access to the secret.

I was not working in ops for a long time and coming back to it, I was surprised to see that this is still an issue. I understand that my work environment on premises and out of cloud is specific and not quite mainstream. On the other hand, take an arbitrary docker compose spec, and it's almost certain that you'll find DB credentials to be set up in environment variables. Certbot using a DNS plugin expects a credentials file. At the same time I keep getting emails informing me that my account has been leaked as part of this or that data leak.

To me it seems to be too expensive for an individual operator or dev to address this problem, because it's more likely that other weaknesses are exploited. But the absence of a design pattern or best practice makes it too hard to close this particular hole.

Answer 1

I have seen an approach that involves two non-secret segments: one in the code, and a second in the configuration file. Each is typically 32 random bytes base64-encoded. These two parts are combined and a key is derived (eg. PBKDF2) which is then used to decrypt the encrypted credentials at runtime. The credentials to be protected might be database password, or an RSA private key, or a passphrase to a key store (a file or a string).

Implications:

1. The developers have access to "segment 1", stored under version control, but not "segment 2" configured by the deployer/admin.

2. An attacker with privileges of the user running the app can read the segment 2 configured value but does not have direct access to the hard-coded segment 1. In the case of a java application, the attacker wold also need to decompile the jar file to find the segment 1 and the key derivation approach.

3. Security analysts generally accept that although an attacker could possibly reassemble the decryption key by discovering both segments, it is more difficult, and more importantly this approach satisfies the audit requirements.

4. For initial deployment, it is necessary to provide the deployer with a script (which contains the hard-coded part 1) so the deployer can provide the configured value (part 2) and the secret to be protected, so that it can be encrypted. Of course, that script must be kept securely elswhere.

This approach can indeed be implemented by an individual developer or project without requiring the entire organisation to agree, and without any special infrastructure.