Managing multiple SSH private keys for a team

Question

I have a medium-sized team (~10 people, but we expect to grow) that manages a set of EC2 servers on AWS (currently a few dozens, but this is also expected to grow).

As EC2 requires you to have an AWS generated private SSH key, without a passphrase (that's how AWS generates them) in order to access EC2 servers, as the number of servers and different environments and products my team manages (and as people are moving on and off the team), I'm less and less happy with our current solution for securing access to EC2 servers - namely having a single private key that is copied to each team members local machine.

I'm considering a few options, and I would appreciate it if you can suggest a better option and discuss why its better:

1. Keep using a single SSH key for all our systems.
   • Pros: simple to manage, relatively secure (assuming team trust)
   • Cons: after a team member gets the key, there is no way to revoke access; a single leak compromises security of all systems; no passwords
2. Have a single SSH key for each product/environment, distribute to all team members.
   • Pros: Still not difficult to manage, relatively secure (assuming team trust)
   • Cons: keys can't be revoked; a single leak compromises all systems (maybe not all, if a junior member that doesn't have all the keys was the source); no passwords; difficult to use as the user has to unload and reload keys when moving between environments
3. Build a bastion server for each product/environment; create a single SSH key for each product/environment and install the private key in the bastion server's known user account; install each team members personal public key in the known user account.
   • Pros: allows key revocation; compromise of a bastion compromises only 1 environment; if a leak of a user key is detected, compromise of untouched system can be easily prevented; allows use of passphrases to access bastions
   • Cons: relatively complex to manage (creating additional servers and running non-trivial installation, adding team members, removing team members); costly (servers aren't free); complicates software tools used by team; complex key revocation
4. Use a key storage service; create a single key for each environment/product and store in service; control access to the service using a password or a personal ssh key; operations start by identifying the environment/product accessed and obtaining the key into the SSH agent.
   • Pros: easy to manage (assuming the service is available); keys can be revoked on a per user or per environment; single point to protect (that is not mission critical); Relies on OpenSSH agent to secure keys outside the service.
   • Cons: Single point of failure; may complicate usage scenario

I'm currently leaning towards #4, but are there serious issues I'm missing? Is there a service like that that I can use or do I have to roll my own?

Note: we don't use an orchestration/configuration server a-la puppet/chef - our orchestration software is mostly home grown and installed on each team member's local system. It is basically just a set of recipes loaded from source control and used to execute various scenarios, mostly using AWS APIs. Each team member has a personal AWS API key and the orchestration software uses it to run the AWS API. In addition, some scenarios call for SSH access to system servers, and here is where I have the problem described above. The EC2 servers are accessed using the default AMI user (usually "ubuntu") and the software uses NOPASSWD sudo to execute local operations.

Answer 1

Not sure why you're using AWS-created keys. The "Network & Security / Key Pairs" screen also has an "Import key pair" button, I've successfully used it.

When you add a new team member who might be standing up new images, get his public key, import it.

Keep a copy of your standard "authorized_keys" file, with one or more public key per person, on a webserver or S3 (could be VPN-facing, or secured by a team-shared username and password, if you're trying to keep your team identities confidential.) Now part of your deploy process is to do

curl -u username:password \
  https://team-server/private/authorized_keys.txt >/home/ec2-user/.ssh/authorized_keys

(Such a command could also be added to the instance user data to run at instance creation.)

To add a new team-member, add his public key to the file, then run that command on all servers.

To remove a team-member, remove his key from the file, then run that command on all servers.

Answer 2

Best practices today require at least considering the use of Bastion hosts or VPN-type network restriction as well as, per the focus of your question, central credential management. I recommend making use of Bastions and using CA-based Key-signing.

Why you should (but don't have to) do bastions anyway

We would like it so when an employee leaves, even if he keeps a copy of his SSH keys, he cannot reach the SSH endpoints. This is a task for bastion hosts, VPNs, and firewalls. Bastion hosts are also a great help in terms of infrastructure in that if you can restrict access to the Bastions (for example by removing the ex-employee's ssh keys) then the user is automatically restricted from accessing the other endpoints.

While Bastion hosts may well make the practice of doing key management easier, it isn't technically essential. Still, I highly recommend making use of bastion hosts: It improves security; simplifies infrastructure; and aids with auditing.

Managing keys

Basic copying of pubkeys into authorized_keys

Copying the public keys as part of an automated task will work. Ansible, SaltStack, cloud-specific deployment tools, cron, and so forth are examples of how that might be implemented. But it doesn't have the obvious advantages of Key-signing below. If you already have something automated that is working for you, it can continue to work well. There are advantages - but you've already put the extra work into making it work, and it isn't broken (right?).

FreeIPA

FreeIPA, if configured on all servers and workstations/laptops, allows you to manage keys centrally. I've found some tools that apparently work with it to support using CA-signed keys, though I'm yet to explore this option.

Key-Signing

Built-in support for signed SSH key certificates has apparently been available in openssh since at least 2010^*, though I've only recently learned of it. SSH key-signing seems to be superior to every other key-management mechanism available. Facebook recently (2016) posted on how they manage certificates - and signing SSH keys is the main ingredient.

The core idea is that you have your own internal CA certify your public keys and have all your ssh endpoints trust the CA. When you connect to an endpoint, ssh presents your public key and the signed certificate, the endpoint verifies that the cert is good, and you are granted access.

Advantages of key-signing:

• Your "personal" keys become ephemeral, meaning you don't have to worry about having a backup of your keys
• Actual keys aren't trusted, meaning compromised keys aren't important
• Certs expire, meaning a compromised key-cert combination has a time limit whereafter it no longer has access to your infrastructure
• Server keys can also be signed, meaning you don't have to manually approve or manage conflicting fingerprints when you connect to signed endpoints (FreeIPA can also handle this, though it technically still does "old-school" copying of the public keys)

How do I use Key signing?

There unfortunately isn't a well-known turn-key system built specifically for orchestrating CA trust and key-signing - but there are tools available you can put together to do the job. For a team, you might be willing to write your own system to sign the keys, though I'm currently looking at a few toolsets. Top among them are HashiCorp's Vault which promises to solve the problem with Signed SSH Certificates using it's Secrets Engine. CommerceHub's ssh-ca-server, promises to allow your CA to sign keys via web interface, whilst using LDAP auth to ensure only valid users' keys are signed. As mentioned, I'm not sure if either of these are full solutions - but they do at least cover a lot of ground. There do seem to be many others as well - though I'm not yet aware of any that seem to be the "best" toolset. See also the informed comments on this answer.

I want to do it manually - or I want to understand how it's done

In a personal capacity, as long as you keep your CA key passphrased, you can use CA signing as below. In a small business setting, I recommend partitioning the key away so no user has direct access to it, or put together existing tools (mentioned above) to do the grunt-work for you. You should also put in place appropriate backup mechanisms - this one key governs access to everything:

Example manual use case

In my example, we're going to use the .ssh/.ca folder for the ca data on the test client. Note of course this is merely an example. The best way to do this is via a remote service:

tricky@testclient $ export CAPATH=~/.ssh/.ca

Generate the CA key pair

You can use rsa for the type - I just prefer ed25519:

tricky@testclient $ ssh-keygen -C CA -f ${CAPATH}/ca -t ed25519
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/tricky/.ssh/.ca/ca.
Your public key has been saved in /home/tricky/.ssh/.ca/ca.pub.
The key fingerprint is:
SHA256:qGSZbssuQOV0TR8WrTV/gzXBSyTO4uAKUO7q+fJUDYw CA
The key's randomart image is:
+--[ED25519 256]--+
|     .o. +o  .oo.|
|   oo.o.o .+o .= |
|  +..E o oo.oo+ o|
| . .oo .+.o .o + |
|.   =o..So .  . .|
|.  +..o .        |
| . .+. .         |
|  o+o.           |
|   =Bo           |
+----[SHA256]-----+
tricky@testclient $ cat ${CAPATH}/ca.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKg9nxnaBLOQKnfuxUQgb3Y1hiqIaMEtXwr8WsX5bLlv CA

Distribute the public part

Distribute the ca.pub file to your remote endpoints in a location accessible to the ssh daemon such as /etc/ssh/ca.pub. This only has to be done once on each server. Make sure it is owned by root and not writeable by non-root users. It is fine to be readable by other users.

Configure the remote endpoint(s)

Add the following line to your sshd_config file on your remote endpoints, referring to where you added the ca.pub file. Again, this only has to be done once on each server:

TrustedUserCAKeys /etc/ssh/ca.pub

Test the config file. You might find "Deprecated option" warnings for unrelated things in the config, especially if you are using a non-stock generated config, but as long as there are no actual errors, you can restart the sshd service:

root@testserver $ sshd -t
/etc/ssh/sshd_config line 23: Deprecated option UsePrivilegeSeparation
/etc/ssh/sshd_config line 26: Deprecated option KeyRegenerationInterval
/etc/ssh/sshd_config line 27: Deprecated option ServerKeyBits
/etc/ssh/sshd_config line 45: Deprecated option RSAAuthentication
/etc/ssh/sshd_config line 54: Deprecated option RhostsRSAAuthentication
root@testserver $ systemctl restart sshd
root@testserver $

Ensure you're not using your old keys

To test, rename your existing keys so you are no longer using explicitly trusted keys:

for type in dsa rsa ed25519 ; do
  mv ~/.ssh/id_${type}{,.bak}
  mv ~/.ssh/id_${type}.pub{,.bak}
done

See notes^** at the end to restore these original keys (after all you will need your existing keys until your CA is trusted by all endpoints).

As a sanity check, if you are using a service such as keychain or ssh-agent that remembers your keys, you might want to stop the service so you know your test is not using your old keys:

tricky@testclient $ killall ssh-agent

Create a new user keypair

You can use rsa for the type - I just prefer ed25519:

tricky@testclient $ ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in testkey.
Your public key has been saved in testkey.pub.
The key fingerprint is:
SHA256:0cKlJxYQrizfd5FDboyVYhn9RK7oHkpsBbarZuv+DoQ tricky@testclient
The key's randomart image is:
+--[ED25519 256]--+
|      ooo....    |
|     . . B.o.    |
|      + @ *o.    |
|   o o = & o.    |
|  E + . S O      |
|   + o + . o     |
|    o * + .      |
|    o= + o       |
|   ==++ .        |
+----[SHA256]-----+
tricky@testclient $

Sign the new user keypair

Keysigning makes use of serial numbers which can be used to aid in auditing. I'm not bothering with recording this in the example. You would at least increment the serial number on subsequent signing events. The serial must be a number:

tricky@testclient $ export serial=1

Identity can theoretically be anything - but client username or user@hostname is a good starting point

tricky@testclient $ export IDENTITY="${USER}@$(hostname)"

The key needs to grant access to specific remote user(s). With a typical bastion configuration, you wouldn't have 'root' here:

tricky@testclient $ export users="root,${USER}"

For a validity period, I generally recommend using 4 hours (14400 seconds) - but you may want a larger time-span. Facebook's post recommends a full day (86400 seconds). Others with automated authentication recommend as little as five minutes.

tricky@testclient $ export VALIDITY_PERIOD=86400

Note that you sign the public key, as it is the public portion that needs to be accepted by the endpoint:

tricky@testclient $ ssh-keygen -s ${CAPATH}/ca -I ${IDENTITY} -n ${users} -V +${VALIDITY_PERIOD} -z ${serial} ~/.ssh/id_ed25519.pub
Enter passphrase:
Signed user key /home/tricky/.ssh/id_ed25519-cert.pub: id "tricky@testclient" serial 1 for root,tricky valid from 2018-10-09T20:06:00 to 2018-10-10T20:06:00

Connect to the endpoint:

tricky@testclient $ ssh root@testserver
Last login: Mon Oct  8 20:46:31 2018 from testclient.localdomain
root@testserver ~ #

---

^* Thanks, @PatrickMevzek, for the ref info:

TrustedUserCAKeys ... was added first in a commit on March 4 2010 openssh.com/txt/release-5.4 "Add support for certificate authentication of users and hosts using a new, minimal OpenSSH certificate format (not X.509)."

^**

for type in dsa rsa ed25519 ; do
  mv ~/.ssh/id_${type}{.bak,}
  mv ~/.ssh/id_${type}.pub{.bak,}
done

Once restored, I recommend signing your existing keys but also to start treating them as ephemeral. Keep a backup of your original keys in case you believe they may be needed for services that trust them but don't support CA-based Key-signing.

Answer 3

After re-reading all this, I insist : having root access granted to many users directly is generally a not so good idea. Use sudo is the recommended way (you could even run su, with group restriction).

Why not use *nix group original behavior?

• For each user, add a specific account ( you could manage user account replication by using LDAP, NIS or even rsync on your passwd, group, shadow and *$USER/.ssh/authorized_keys by a simple shell script... NIS is nice! )

  adduser alice
  

• Add specific passwd to each user (user may or need to change them... or not)

  passwd alice
  

• Add user to group with specific right (to acccess sudo, for sample)

  adduser alice sudo
  

• Doing so, you could create another group with common right, for working together on specific project, even without access to root.

Doing this way let the ability to

• let user connect without passwd, but only on his account. Then ask user for password when they run sudo.

  su - alice <<<eocmd
     curl -u username:password \
        https://team-server/private/authorized_keys.txt >~/.ssh/authorized_keys
  eocmd
  

• add or delete as many user you want

  deluser alice
  

• drop or restore right to

  • access to any host/server

    usermod --expiredate 1 alice
    

  • access to specific group rights.

    deluser alice sudo
    

• keep specific authorized_keys for each user, for sample

  su - alice -c 'curl ... >>.ssh/authorized_keys'
  

  to permit one user to connect from many different points. In fact, as each authorized_key file is owned by his user, each could do what he want, without compromising whole host.

... and keep PermitRootLogin to no in /etc/ssh/sshd_config!

Answer 4

I believe you are either misunderstanding or over-estimating the complexity of a bastion host.

bastion server for each product/environment ... you should only need 1x bastion host (2x for redundancy just in case).

running non-trivial installation ... if you have OpenSSH that is already a bastion host (you would just harden it as you would any other externally facing server)

I currently run a bastion host for my personal setup, here is what I do:

Bastion host - VPS (costs me $2.50 a month 1cpu 512mb 500gb/m ymmv) it has OpenSSH, UFW, Fail2Ban

Internal[0-7] - various VPS & Physical servers I manage for my friends and family ... each has OpenSSH configured to only accept connections from Bastion host

Bastion host's authorized_keys (personally moved it out of user profile so there is 1 file to rule them all and so non-admin users cant modify it)

ssh-rsa AAAA ... 4W95 myAdminUser@desktop
ssh-rsa AAAA ... rHjK alice
ssh-rsa AAAA ... DkQP bob

you could disable shell on a per key for non-admin users ... I personally trust my users to have non-root shell on the bastion host ymmv

Internal[0-7]'s /etc/ssh/sshd_config

...
AllowUsers *@<bastion.host.static.ip>
...

Desktop's ~/.ssh/config:

Host bh
  HostName bastion.host.domain.name
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlMaster auto
  ControlPersist 10m

Host intN
  HostName n.internal.domain.name
  User <username>
  IdentityFile ~/.ssh/id_rsa
  ProxyCommand ssh -W %h:%p bh

Using this ~/.ssh/config you would use the command ssh intN to connect ... alternitivly ssh user1@intN and scp intN:/foo/bar ./ also work

when someone wants access, I tell them to give me a username and a pub-key, an account is added to each server they should have access to with pub-key added to there users ~/.ssh/authorized_keys (I personally automate this w/ github's ssh-key api and Ansible ymmv)

when someone no-longer needs access I comment out # there pub-key from the authorized_keys list on that specific box

when someone no-longer needs access to anything I comment out # there pub-key from the authorized_keys list on the bastion host ... even if there key is left on the individual servers ... if they cant get into the bastion host they cant get in to anything.

Answer 5

Maybe a solution such as FreeIPA could help you. It provides a way to make sure that ssh keys are properly distributed as well as sudo profiles and authentication.

Z.

Answer 6

A typical best-practices answer would be to use something like LDAP. With it, you can define users, groups, and more.

By connecting your SSH authentication backend via PAM to your LDAP server, you can maintain separate accounts for each member of your team. I may be mistaken, but you should also be able to maintain separate $HOME/.ssh/authorized_keys files for each user as well.

By making users members of the sudoers group, they'll be able to request root permissions on-demand. These accesses will be recorded in logs, so you'll have auditability later on down the line.

This setup is pretty standard in larger corporations.

Answer 7

When I need to distribute public ssh-keys on my servers, I use an online tool (https://keydistributor.io) and fetch the keys on each server from their API. In that tool (just like Puppet, chef etc) I could define groups, where each user belongs to a group.

Answer 8

A simpler way is as shown below.

For Linux / Mac users:

To create Public and Private keys use the following command: ssh-keygen -t rsa -b 4096

Upload the public key to a folder in your S3 bucket. For example: S3 > MyBucket > Keypair

Save and secure your private key.

For Windows users:

Use puttygen to generate the keys. Upload the public key to S3 > MyBucket > Keypair

Save and secure your private key. The following steps are important during the launch of any Linux AMI.

Ensure the IAM role has a role created with AmazonS3FullAccess policy. This allows the instance to assume a role to access the S3 buckets. This is needed to read the public keys from S3 and copy them to the user profile.

Add the following code under the user-data section in Configure Instance details > Advanced Details (as Text):

    #!/bin/bash
    usermod user 1
    usermod -aG wheel user1
    mkdir /home/user1/.ssh/
    aws s3 cp s3://MyBucket    /Keypair/user1-pub.pub /home/user1/.ssh/authorized_keys

    useradd user2
    usermod -aG wheel user2
    mkdir /home/user2/.ssh/
    aws s3 cp s3://MyBucket   /Keypair/user2-pub.pub /home/user2/.ssh/authorized_keys

    sudo -i 
    echo “user1 ALL=(ALL) NOPASSWD:ALL” >> /etc/sudoers
    echo “user2 ALL=(ALL) NOPASSWD:ALL” >> /etc/sudoers

yum update -y

This setup creates User1 and User2 and adds them to sudo users. The aws s3 cp command copies the users public keys from the S3 folder to their .ssh/authorized_keys path. The last section is to run commands as admin without needing passwords.

There are lots of security improvements that can be recommended here. While not explicitly used in this example, limiting S3 bucket access to a specific bucket and knowing the security implications of disabling password usage in sudo, are few things that can be highlighted. Use them wisely based on your particular needs.

Answer 9

Opera Software's SKA - SSH Key Authority offers an interesting solution: as far as I understand it is a central server that allows you to register servers and setup rules for how users (either manually managed or synced from LDAP) have access or admin rights in registered servers, then users can upload their public keys and SKA publishes keys to servers by SSHing in as a pre-prepared "sync-keys" users and updating a set of SKA specific files that sshd has to be configured to read.

On the face of it I don't like it because:

• servers have to be registered ahead of time. Setting up automatic registration (and worse - automatic deregisteration) will be quite a hassle.
  • I'm not happy about the requirement to configure all servers with a specific user and configure sshd in a non default way, that will break access for the cloud platform's default user account (so if the configuration breaks, there's no way to fix it or figure out why it broke).

Otherwise, it looks very promising, especially the semi-automatic team management and logging & notifications.

Answer 10

If you don't want a key distribution machinery then using a SSH-CA for issuing tempory OpenSSH certificates might be the right choice.

With this you only have to install the SSH-CA's public key as trust anchor on the target machines (see sshd_config directive TrustedUserCAKeys).

Caveat: This currently only works with OpenSSH and libssh.