Is it unsafe to use environmental variables for secret data?

Question

After reading this blog post in which the author lays out arguments against using environmental variables for storing secrets, I am unsure how to proceed with deploying my application.

His primary arguments are as follows:

• Given that the environment is implicitly available to the process, it's hard, if not impossible, to track access and how the contents get exposed (ps -eww <PID>).

• It's common to have applications grab the whole environment and print it out for debugging or error reporting. So many secrets get leaked to PagerDuty that they have a well-greased internal process to scrub them from their infrastructure.

• Environment variables are passed down to child processes, which allows for unintended access. This breaks the principle of least privilege. Imagine that as part of your application, you call to a third-party tool to perform some action—all of a sudden that third-party tool has access to your environment, and god knows what it will do with it.

• When applications crash, it's common for them to store the environment variables in log-files for later debugging. This means plain-text secrets on disk.

• Putting secrets in ENV variables quickly turns into tribal knowledge. New engineers who are not aware of the sensitive nature of specific environment variables will not handle them appropriately/with care (filtering them to sub-processes, etc).

These seem soundly reasonable to me, but I am not a security professional. His alternative suggestion is to use Docker's secret-keeping functionality, but that's assuming that you're using Docker....which I'm not. I'm on Heroku. So I'm kind of unsure about this now. There doesn't seem to be any support for using Vault on Heroku, as best as I can tell.

Answer 1

In general storing secrets in environment variables does have some downsides, as Diogo says in his post.

Generically, for platforms like Heroku, or using technologies like Docker where the application is expected to be ephemeral, dedicated secrets management tools are the best way to go. The idea is that there should be a tool which holds the secret in encrypted form and provides it to the application at runtime.

The secret can then live inside the application, generally as a file, which can be read and the secrets used from that file as needed.

Two examples of tools in this area are Hashicorp Vault and Square's Keywhiz.

In addition to this if you're deploying on a cloud, generally the cloud provider should have some kind of secrets management facility, for example AWS Secrets Manager.

I've not had much experience with secrets management on Heroku, however they do seem to have an add-on called ice which operates in this area.

Answer 2

If you're fine with the security properties of typical key-management solutions (like the one built into Docker that's mentioned in the article, or the ones mentioned in the other answer) then you can use environment variables to pass secrets without any of the mentioned downsides, provided you immediately delete them from the environment once you've gotten them.

During early initialization, before you run libraries or programs which you didn't audit, read the secret from the environment variable into a normal variable in your program, and then delete it from the environment (for example, in C you would use unsetenv("SECRET"), in Python you would use del os.environ['SECRET'], and so on).

Similarly, if your code is passing a secret through an environment variable to another process, make sure the environment variable is no longer set after you've started the process that needed the secret.

This eliminates risks like accidentally leaking the secret to malicious/sloppy code that you call, into your logging messages, and so on.

To be even more paranoid, you may want to override the environment variable's old bytes in-place, to protect against code with raw memory access trying to find it in that relatively known/fixed location of arguments and environment variables passed to the process at startup. On every modern OS that I've checked, the initial environment variable memory passed to the process is mutable, so in something like C you can literally just start writing through the pointer to that environment variable. In most other languages, if you want to get that secure, you'll probably need to link a library that does it with native code.

The only risk that remains after you've scrubbed the secret from the environment is that processes on the same machine which are running as your user (or as a privileged user) could've read it in the time window between it being set and it being scrubbed. This is safer than what everyone seems to consider acceptable with all the secret manager solutions. When Docker writes a secret to a container's /run/secrets/, everything in that container (which hasn't been deliberately privilege-dropped) can read it. When AWS gives a VM access to an entry in Secrets Manager, basically anything on that machine has access it.

And if you want something even more secure-by-default for passing secrets between processes, you should pass secrets in on standard input.