What are some best practices for managing and deploying web app passwords and private keys? E.g. the password to access a database or external services like the Facebook API. I'm referring generally to using dedicated hardware or a general cloud provider and not any specific provider like Heroku or EC2.
Is the best you can do without an HSM is to deploy the credentials in a plain text file to a read protected directory that can be accessed only by the web app user? Should you have one file with all passwords or a file per user or store in environment variables? How do you manage actually putting them on the servers? Can you use a configuration management tool (e.g. chef/puppet) to deploy them and keep them up-to-date? How should these credentials be stored? Should they be stored in encrypted format in the source control directory containing the configuration management configs? Any recommendations on tools to use that may be helpful? (e.g. keyczar, secret server, etc.) If the passwords are encrypted what's a good way to share the decryption key among the DevOps team? (LastPass, KeePass, etc.?)
I know that's a long list of questions, but these are all things that I haven't found good answers to. I'm interested in any info whether that's a case study of someone doing it well, trade-offs to be made between different alternatives, things that should be clearly avoided or aimed for. I really don't know what best practices exist and have helpful tools vs. what we just have to decide for our own situation and solve on our own.