There seems to be a general recommendation to store secrets in the Hashicorp Vault instance (or similar key-management software) and avoid passing secrets via environment variables. In what particular scenarios using Vault is better from security point of view than using environment variables?
Vault's promise is "secrets as a service". It supports static storage of secrets (think encrypted Redis/Memcached), pass-through encryption (give Vault plaintext, vault gives back ciphertext that you store in a database), and dynamic secret acquisition.
On the static secret side of things, data is encrypted in transit and at rest. Data can be stored in memory, on the file system, or in third-party tools like Etcd or Consul. This is great for application-level secrets. Vault supports online rotation of the underlying encryption key. If you have FIPS/HIPPA/PCI compliance requirements, Vault makes it easy to check off most of those boxes with the default configuration.
With the pass-through encryption (or "transit" as it's called internally), Vault acts as an encryption service, accepting plaintext data, encrypting it, and returning the ciphertext. I wrote about this process in much more detail on the HashiCorp blog, but the process is simple. This ciphertext is then managed by your application. When the application needs the plaintext back, it authenticates and authorized to Vault, provides Vault the ciphertext, and Vault returns the plaintext (again, if authorized). There are a ton of benefits here, but the biggest ones are: 1. You don't have to build a symmetric encryption service into your application; just make an API call, and 2. The encryption keys are stored in a completely separate and isolated service; if an attacker must compromise multiple systems. Additionally, Vault's transit backend supports this concept called "derived keys". This enables things like per-row encryption keys for data stored in a database, such that even if an attacker had a database dump and could brute force the first encryption key, that key would not decrypt the other rows in the database. Just like the static secret backend, the transit backend supports key rotation.
The dynamic secret backends, in my opinion, are where Vault truly separates itself from other or home-grown solutions. Vault can connect to and dynamically generate credentials from things like databases, cloud credentials, CA certificates, manage SSH access, and more. Unlike traditional credentials, these credentials have a lease associated with them, akin to something like DNS or DHCP. When an application is given a credential, it's also given a "lease" or lifetime of that credential. Over time, the application (or a service) must communicate with Vault that it's still using that credential or Vault will revoke it. This helps eliminate secret sprawl while still providing a programatic way to access credentials. Since this is programatic, each instance of the application (or python script in your example) receives a different secret. You can easily revoke a single application's credentials without affecting the entire system.
Here are some use cases that might help you get buy-in internally:
Use Vault's GitHub authentication to authenticate your developers and operators. GitHub team membership is mapped to policy in Vault. Anyone in the ops team gets SSH access to prod, and anyone in the dev team gets the ability to generate dynamic AWS account credentials in the staging environment for testing.
Use Vault's AppRole authentication to have applications authenticate to Vault and retrieve a token. From there, the application's policy allows it to retrieve startup data, such as a database credential. If the app crashes, the database credential is automatically revoked when the lease expires.
You've detected a data leak. You can immediately revoke all affected credentials. This is called a "break glass" procedure.
As an additional note, you can use a tool like Consul Template to pull values from Vault into a template that your application can then consume. Your application does not need to be "Vault aware".
And finally, probably not related to your problem given the post, but it's worth pointing out that Vault also solves the "no one person has complete access to the system" challenge that most organizations face. By using Shamir's Secret Sharing algorithm, the process for bringing a Vault server online is very similar to unlocking a traditional bank Vault - multiple people must enter their key simultaneously to unlock. You can also read more about Vault's security model.
For videos that have demos and stuff, check out: