In an extremely interesting presentation at Puppet Camp London, Tomas Doran suggested a pretty radical approach for keeping everything automated by managing tons of Docker containers with Puppet.

As a security-conscious person, I like the idea of Docker, as everything runs in its own LXC and I configure all services to run as non-root users.

As a systems-administration-conscious person, I like the idea of Puppet management, as I can keep all configuration in a Git repository and even maintain different environments, all in version control. The advantage is also that I have tear-down-able (is that a word?) environments that I can theoretically rebuild from scratch without too much manual intervention.

However, there are things I'd like not to keep in a Git repository, namely SSL certificates, database passwords, etc.

How do organizations managing massive amounts of machines (like CERN) use provisioning services like Puppet and Chef while still maintaining security? Certain things seem easy, like enforcing permissions on files, but other things seem difficult, like installing SSL keys or SSH host keys, which requires manual intervention.


I used to run a puppet installation, so I'll speak to what I did then.

  1. SSH Host Keys are ephemeral. So, when building a new system or rebuilding an existing one, we just let new keys be generated by SSHD when it first started. There were relatively few people SSHing into these machines, and when rebuilding the machines, we published the host key fingerprints on an internal wiki. So an attacker would have to compromise the wiki host and MITM SSH connections, and be in place to do so immediately after the machine was built.

  2. We always generated new SSL keys when reinstalling the machine. One of the human steps in rebuilding services that involved SSL was generating the key, the CSR, and getting it signed by the CA and then installing the cert. We did use a script that was prepopulated by a puppet template to have the CN & other fields properly set.

I doubt CERN has a lot of hosts with SSL certificates involved. Or maybe they do, but have an automated CA to sign certs at puppet's request.

I also don't consider storing certs & keys in a well-controlled git repository necessarily bad. We just took the point at which we were rebuilding a system as a great opportunity to rotate our keys.

  • It looks like you missed a great opportunity to use SSHFP DNS records to publish your public keys. Jun 19 '14 at 12:57
  • SSHFP is only useful if you have DNSSEC. Without DNSSEC, you can't know if the SSHFP record is valid. (Though, admittedly, MITMing DNS is probably comparable to breaking our wiki...)
    – David
    Jun 19 '14 at 16:18
  • IIRC if you don't have DNSSEC (or to be pedantic, if the reply isn't authenticated) it still informs you with the message "Matching fingerprint found in DNS" in addition to presenting you the fingerprint and the yes/no question. So even without DNSSEC it brings some level of convenience. Jun 19 '14 at 17:41
  • jpmens.net/2012/07/27/verifyhostkeydns-yessssss explains pretty well how it works in both cases, with example outputs Jun 19 '14 at 17:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.