36

After reading this blog post in which the author lays out arguments against using environmental variables for storing secrets, I am unsure how to proceed with deploying my application.

His primary arguments are as follows:

  • Given that the environment is implicitly available to the process, it's hard, if not impossible, to track access and how the contents get exposed (ps -eww ).

  • It's common to have applications grab the whole environment and print it out for debugging or error reporting. So many secrets get leaked to PagerDuty that they have a well-greased internal process to scrub them from their infrastructure.

  • Environment variables are passed down to child processes, which allows for unintended access. This breaks the principle of least privilege. Imagine that as part of your application, you call to a third-party tool to perform some action—all of a sudden that third-party tool has access to your environment, and god knows what it will do with it.

  • When applications crash, it's common for them to store the environment variables in log-files for later debugging. This means plain-text secrets on disk.

  • Putting secrets in ENV variables quickly turns into tribal knowledge. New engineers who are not aware of the sensitive nature of specific environment variables will not handle them appropriately/with care (filtering them to sub-processes, etc).

These seem soundly reasonable to me, but I am not a security professional. His alternative suggestion is to use Docker's secret-keeping functionality, but that's assuming that you're using Docker....which I'm not. I'm on Heroku. So I'm kind of unsure about this now. There doesn't seem to be any support for using Vault on Heroku, best I can tell.

4
  • I'm pretty sure this is a duplicate. Tl;dr though, there are more secure ways to pass secret data, but it's not quite that horrible (e.g. you can't access /proc/<pid>/environ without running as the same user).
    – forest
    Nov 16, 2018 at 8:30
  • // , It's a duplicate, but of an SO question: stackoverflow.com/a/4136344/2146138 seems to suggest that, assuming an application's user is the only one that can read a configuration file, using environment variables isn't that much more secure than putting the secrets into that configuration file with proper Linux file protections in place. Duplicate or not, it's a story worth the telling to hear. Jan 11, 2019 at 2:47
  • // , I asked about a specific case of "put secrets in environment variables or a properly protected file" here: security.stackexchange.com/questions/201245/… Jan 11, 2019 at 2:48
  • I agree it isn't MUCH more secure than a config file - but with the caveat that this assumes the bad actor has access to the environment in the first place. The worry with config files is that they are so easily accidentally included in version control. That makes it darn-near impossible to clean up because anybody could have a copy, even former employees.
    – Sam Heuck
    Feb 15 at 16:19

2 Answers 2

14

In general storing secrets in environment variables does have some downsides, as Diogo says in his post.

Generically, for platforms like Heroku, or using technologies like Docker where the application is expected to be ephemeral, dedicated secrets management tools are the best way to go. The idea is that there should be a tool which holds the secret in encrypted form and provides it to the application at runtime.

The secret can then live inside the application, generally as a file, which can be read and the secrets used from that file as needed.

Two examples of tools in this area are Hashicorp Vault and Square's Keywhiz.

In addition to this if you're deploying on a cloud, generally the cloud provider should have some kind of secrets management facility, for example AWS Secrets Manager.

I've not had much experience with secrets management on Heroku, however they do seem to have an add-on called ice which operates in this area.

6
  • I'm having a bit of an issue comprehending how these services help. Firstly, as best I can tell, all ICE does is generate credentials to use with AWS KMS. Sure, that's fine, I guess. But then here's the way I'm thinking about it: let's say I follow the Ruby example listed in the ICE documentation. (cont'd) Nov 17, 2018 at 3:45
  • 2
    (cont'd) Excellent, now my credentials are encrypted. But....the ICE credentials are still stored in plaintext as environmental variables, as demonstrated in that code. So a malicious piece of code could just access them and then use AWS KMS to decrypt my actual secrets the same way I'm doing. So how is anything really changed here? Nov 17, 2018 at 3:45
  • @temporary_user_name. I agree. The machine that ends up using the credentials needs to be authenticated. Proxying that authentication via e.g. AWS Secrets manager does not make it more secure. However: if the credentials are compromised, it can be as easy as a 1-click to create new credentials if automated well. You then need to change only one password/keypair as opposed to quite a lot.
    – marstato
    Dec 28, 2018 at 10:43
  • // , Hell yes, Rory. HashiCorp Vault is the more "enterprise-ey" of the two, and I have personally used it to solve this problem on a global scale for RSA keys. Thanks for not only answering the question, but also including possible solutions to the attendant problems. Jan 14, 2019 at 21:14
  • 1
    What about storing secrets in .env.developement (for testing purposes, e.g. pre-filled password fields for recurring login testing)? Given that this .env file is only used locally and not commited to the repository.
    – elMeroMero
    Oct 28, 2020 at 13:13
4

If you're fine with the security properties of typical key-management solutions (like the one built into Docker that's mentioned in the article, or the ones mentioned in the other answer) then you can use environment variables to pass secrets without any of the mentioned downsides, provided you immediately delete them from the environment once you've gotten them.

During early initialization, before you run libraries or programs which you didn't audit, read the secret from the environment variable into a normal variable in your program, and then delete it from the environment (for example, in C you would use unsetenv("SECRET"), in Python you would use del os.environ['SECRET'], and so on).

Similarly, if your code is passing a secret through an environment variable to another process, make sure the environment variable is no longer set after you've started the process that needed the secret.

This eliminates risks like accidentally leaking the secret to malicious/sloppy code that you call, into your logging messages, and so on.

To be even more paranoid, you may want to override the environment variable's old bytes in-place, to protect against code with raw memory access trying to find it in that relatively known/fixed location of arguments and environment variables passed to the process at startup. On every modern OS that I've checked, the initial environment variable memory passed to the process is mutable, so in something like C you can literally just start writing through the pointer to that environment variable. In most other languages, if you want to get that secure, you'll probably need to link a library that does it with native code.

The only risk that remains after you've scrubbed the secret from the environment is that processes on the same machine which are running as your user (or as a privileged user) could've read it in the time window between it being set and it being scrubbed. This is safer than what everyone seems to consider acceptable with all the secret manager solutions. When Docker writes a secret to a container's /run/secrets/, everything in that container (which hasn't been deliberately privilege-dropped) can read it. When AWS gives a VM access to an entry in Secrets Manager, basically anything on that machine has access it.

And if you want something even more secure-by-default for passing secrets between processes, you should pass secrets in on standard input.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .