31

I have a medium-sized team (~10 people, but we expect to grow) that manages a set of EC2 servers on AWS (currently a few dozens, but this is also expected to grow).

As EC2 requires you to have an AWS generated private SSH key, without a passphrase (that's how AWS generates them) in order to access EC2 servers, as the number of servers and different environments and products my team manages (and as people are moving on and off the team), I'm less and less happy with our current solution for securing access to EC2 servers - namely having a single private key that is copied to each team members local machine.

I'm considering a few options, and I would appreciate it if you can suggest a better option and discuss why its better:

  1. Keep using a single SSH key for all our systems.
    • Pros: simple to manage, relatively secure (assuming team trust)
    • Cons: after a team member gets the key, there is no way to revoke access; a single leak compromises security of all systems; no passwords
  2. Have a single SSH key for each product/environment, distribute to all team members.
    • Pros: Still not difficult to manage, relatively secure (assuming team trust)
    • Cons: keys can't be revoked; a single leak compromises all systems (maybe not all, if a junior member that doesn't have all the keys was the source); no passwords; difficult to use as the user has to unload and reload keys when moving between environments
  3. Build a bastion server for each product/environment; create a single SSH key for each product/environment and install the private key in the bastion server's known user account; install each team members personal public key in the known user account.
    • Pros: allows key revocation; compromise of a bastion compromises only 1 environment; if a leak of a user key is detected, compromise of untouched system can be easily prevented; allows use of passphrases to access bastions
    • Cons: relatively complex to manage (creating additional servers and running non-trivial installation, adding team members, removing team members); costly (servers aren't free); complicates software tools used by team; complex key revocation
  4. Use a key storage service; create a single key for each environment/product and store in service; control access to the service using a password or a personal ssh key; operations start by identifying the environment/product accessed and obtaining the key into the SSH agent.
    • Pros: easy to manage (assuming the service is available); keys can be revoked on a per user or per environment; single point to protect (that is not mission critical); Relies on OpenSSH agent to secure keys outside the service.
    • Cons: Single point of failure; may complicate usage scenario

I'm currently leaning towards #4, but are there serious issues I'm missing? Is there a service like that that I can use or do I have to roll my own?

Note: we don't use an orchestration/configuration server a-la puppet/chef - our orchestration software is mostly home grown and installed on each team member's local system. It is basically just a set of recipes loaded from source control and used to execute various scenarios, mostly using AWS APIs. Each team member has a personal AWS API key and the orchestration software uses it to run the AWS API. In addition, some scenarios call for SSH access to system servers, and here is where I have the problem described above. The EC2 servers are accessed using the default AMI user (usually "ubuntu") and the software uses NOPASSWD sudo to execute local operations.

  • So screw the whole idea behind SSH's decentralized key management. You want key escrow? – jas- Aug 26 '15 at 3:57
  • Yes, something like that. I want to use personalized access to a key escrow vaule :) Decentralized key management works great for n:1 relationships, or even 1:n, but it gets really messy when you have an n:n setup. – Guss Aug 26 '15 at 8:46
  • 1
    Everything gets messy with n:n – jas- Aug 27 '15 at 2:46
  • Please clarify there is no way to revoke access . In fact, EC2 ssh key can be replaced. – mootmoot Oct 10 '18 at 8:26
  • @mootmoot - I may have exaggerated, but generally speaking, replacing the AWS public key on a running server leads to all kinds of chaos, and I'm treating EC2 instances as immutable - changes are performed by replacing the instance. Just recently we had a team member leave immediately after we completed ISO 27K certification and we had a chance to excercise key replacement - it requires updating the cloudformation stacks then recreating all the servers on all stacks. Now try to do that monthly without downtime. Not impossible but tedious and error prone. The response time to a leak is horrible – Guss Oct 10 '18 at 8:31

10 Answers 10

17

Not sure why you're using AWS-created keys. The "Network & Security / Key Pairs" screen also has an "Import key pair" button, I've successfully used it.

When you add a new team member who might be standing up new images, get his public key, import it.

Keep a copy of your standard "authorized_keys" file, with one or more public key per person, on a webserver or S3 (could be VPN-facing, or secured by a team-shared username and password, if you're trying to keep your team identities confidential.) Now part of your deploy process is to do

curl -u username:password \
  https://team-server/private/authorized_keys.txt >/home/ec2-user/.ssh/authorized_keys

(Such a command could also be added to the instance user data to run at instance creation.)

To add a new team-member, add his public key to the file, then run that command on all servers.

To remove a team-member, remove his key from the file, then run that command on all servers.

  • Ok, so the setup will be to import all members' public keys to Amazon, and when a deployment scenario is played out, the orchestration software will need to choose the correct key (for the member running the scenario) to use in the AWS API call to start the server, then log in and load the rest of the keys? I think its doable. – Guss May 4 '15 at 20:53
  • 1
    How do you actually start the deployment scenario? If the member running the scenario directly initiates it from the command line, the orchestration software may not need to do anything - simply use SSH agent forwarding. Otherwise, you run into the issue that your orchestration server may need the team member's private key (a big no-no in SSH security). – Kevin Keane May 5 '15 at 3:23
  • Ah, sorry it wasn't clear - there is no orchestration server. The orchestration software is installed on each team member's local system. I've updated my question to reflect that. – Guss May 5 '15 at 15:31
  • Wouldn't the authorized keys file get wiped if that web server goes down? That'd leave me locked out. – Ben Poulson Jun 12 '17 at 13:46
  • Agreed with @BenPoulson that the "download" command should verify the new version of the authorized_keys file is complete rather than blindly overwrite the existing copy. The chance of this causing an issue isn't so bad (after all, the web server has to be down when you run the command) - but the risk is bad enough. – zaTricky Oct 9 '18 at 9:42
5

After re-reading all this, I insist : having root access granted to many users directly is generally a not so good idea. Use sudo is the recommended way (you could even run su, with group restriction).

Why not use *nix group original behavior?

  • For each user, add a specific account ( you could manage user account replication by using LDAP, NIS or even rsync on your passwd, group, shadow and *$USER/.ssh/authorized_keys by a simple shell script... NIS is nice! )

    adduser alice
    
  • Add specific passwd to each user (user may or need to change them... or not)

    passwd alice
    
  • Add user to group with specific right (to acccess sudo, for sample)

    adduser alice sudo
    
  • Doing so, you could create another group with common right, for working together on specific project, even without access to root.

Doing this way let the ability to

  • let user connect without passwd, but only on his account. Then ask user for password when they run sudo.

    su - alice <<<eocmd
       curl -u username:password \
          https://team-server/private/authorized_keys.txt >~/.ssh/authorized_keys
    eocmd
    
  • add or delete as many user you want

    deluser alice
    
  • drop or restore right to

    • access to any host/server

      usermod --expiredate 1 alice
      
    • access to specific group rights.

      deluser alice sudo
      
  • keep specific authorized_keys for each user, for sample

    su - alice -c 'curl ... >>.ssh/authorized_keys'
    

    to permit one user to connect from many different points. In fact, as each authorized_key file is owned by his user, each could do what he want, without compromising whole host.

... and keep PermitRootLogin to no in /etc/ssh/sshd_config!

  • Read The Linux System Administrator's Guide for further docs. – F. Hauri May 5 '15 at 8:35
  • We are definitely not logging in as root, sorry it wasn't clear - I've updated my question with details. – Guss May 5 '15 at 15:40
  • If I understand correctly, what you suggest is to either use a central login service (which will need to be managed and protected - I'm not happy doing that once, but as we are using VPC and there is no way that I will put this server on the public internet, we'll need one for each environment), or manage multiple users and multiple SSH keys on multiple servers on a remote "farm" (multiples of those as we plan to expand to multiple AWS sites). Looks like way too heavy lifting. Also, requiring passwords on the each system is a big no-no as it will break our orchestration software. – Guss May 5 '15 at 15:44
  • Hem yes, even if you need to keep root access without password for administrative scripts, having separated account for each sysadmin is mostly a good practice. Of couse, each sysadmin who have access to sudo could install his own backdoor. You have to keep this in mind. – F. Hauri May 5 '15 at 15:59
3

Best practices today require at least considering the use of Bastion hosts or VPN-type network restriction as well as, per the focus of your question, central credential management. I recommend making use of Bastions and using CA-based Key-signing.

Why you should (but don't have to) do bastions anyway

We would like it so when an employee leaves, even if he keeps a copy of his SSH keys, he cannot reach the SSH endpoints. This is a task for bastion hosts, VPNs, and firewalls. Bastion hosts are also a great help in terms of infrastructure in that if you can restrict access to the Bastions (for example by removing the ex-employee's ssh keys) then the user is automatically restricted from accessing the other endpoints.

While Bastion hosts may well make the practice of doing key management easier, it isn't technically essential. Still, I highly recommend making use of bastion hosts: It improves security; simplifies infrastructure; and aids with auditing.

Managing keys

Basic copying of pubkeys into authorized_keys

Copying the public keys as part of an automated task will work. Ansible, SaltStack, cloud-specific deployment tools, cron, and so forth are examples of how that might be implemented. But it doesn't have the obvious advantages of Key-signing below. If you already have something automated that is working for you, it can continue to work well. There are advantages - but you've already put the extra work into making it work, and it isn't broken (right?).

FreeIPA

FreeIPA, if configured on all servers and workstations/laptops, allows you to manage keys centrally. I've found some tools that apparently work with it to support using CA-signed keys, though I'm yet to explore this option.

Key-Signing

Built-in support for signed SSH key certificates has apparently been available in openssh since at least 2010*, though I've only recently learned of it. SSH key-signing seems to be superior to every other key-management mechanism available. Facebook recently (2016) posted on how they manage certificates - and signing SSH keys is the main ingredient.

The core idea is that you have your own internal CA certify your public keys and have all your ssh endpoints trust the CA. When you connect to an endpoint, ssh presents your public key and the signed certificate, the endpoint verifies that the cert is good, and you are granted access.

Advantages of key-signing:

  • your "personal" keys become ephemeral - you don't have to worry about having a backup of your keys
  • actual keys aren't trusted, meaning compromised keys aren't important
  • certs expire, meaning a compromised key-cert combination has a time limit whereafter it no longer has access to your infrastructure
  • server keys can also be signed, meaning you don't have to manually approve or manage conflicting fingerprints when you connect to signed endpoints (FreeIPA can also handle this, though it technically still does "old-school" copying of the public keys)

How do I use Key signing?

There unfortunately isn't a well-known turn-key system built specifically for orchestrating CA trust and key-signing - but there are tools available you can put together to do the job. For a team, you might be willing to write your own system to sign the keys, though I'm currently looking at a few toolsets. Top among them are HashiCorp's Vault which promises to solve the problem with Signed SSH Certificates using it's Secrets Engine. CommerceHub's ssh-ca-server, promises to allow your CA to sign keys via web interface, whilst using LDAP auth to ensure only valid users' keys are signed. As mentioned, I'm not sure if either of these are full solutions - but they do at least cover a lot of ground. There do seem to be many others as well - though I'm not yet aware of any that seem to be the "best" toolset. See also the informed comments on this answer.

I want to do it manually - or I want to understand how it's done

In a personal capacity, as long as you keep your CA key passphrased, you can use CA signing as below. In a small business setting, I recommend partitioning the key away so no user has direct access to it, or put together existing tools (mentioned above) to do the grunt-work for you. You should also put in place appropriate backup mechanisms - this one key governs access to everything:

Example manual use case

In my example, we're going to use the .ssh/.ca folder for the ca data on the test client. Note of course this is merely an example. The best way to do this is via a remote service:

tricky@testclient $ export CAPATH=~/.ssh/.ca

Generate the CA key pair

You can use rsa for the type - I just prefer ed25519:

tricky@testclient $ ssh-keygen -C CA -f ${CAPATH}/ca -t ed25519
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/tricky/.ssh/.ca/ca.
Your public key has been saved in /home/tricky/.ssh/.ca/ca.pub.
The key fingerprint is:
SHA256:qGSZbssuQOV0TR8WrTV/gzXBSyTO4uAKUO7q+fJUDYw CA
The key's randomart image is:
+--[ED25519 256]--+
|     .o. +o  .oo.|
|   oo.o.o .+o .= |
|  +..E o oo.oo+ o|
| . .oo .+.o .o + |
|.   =o..So .  . .|
|.  +..o .        |
| . .+. .         |
|  o+o.           |
|   =Bo           |
+----[SHA256]-----+
tricky@testclient $ cat ${CAPATH}/ca.pub
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKg9nxnaBLOQKnfuxUQgb3Y1hiqIaMEtXwr8WsX5bLlv CA

Distribute the public part

Distribute the ca.pub file to your remote endpoints in a location accessible to the ssh daemon such as /etc/ssh/ca.pub - this only has to be done once on each server. Make sure it is owned by root and not writeable by non-root users. It is fine to be readable by other users.

Configure the remote endpoint(s)

Add the following line to your sshd_config file on your remote endpoints, referring to where you added the ca.pub file. Again, this only has to be done once on each server:

TrustedUserCAKeys /etc/ssh/ca.pub

Test the config file. You might find "Deprecated option" warnings for unrelated things in the config, especially if you are using a non-stock generated config - but as long as there are no actual errors, you can restart the sshd service:

root@testserver $ sshd -t
/etc/ssh/sshd_config line 23: Deprecated option UsePrivilegeSeparation
/etc/ssh/sshd_config line 26: Deprecated option KeyRegenerationInterval
/etc/ssh/sshd_config line 27: Deprecated option ServerKeyBits
/etc/ssh/sshd_config line 45: Deprecated option RSAAuthentication
/etc/ssh/sshd_config line 54: Deprecated option RhostsRSAAuthentication
root@testserver $ systemctl restart sshd
root@testserver $

Ensure you're not using your old keys

To test, rename your existing keys so you are no longer using explicitly trusted keys:

for type in dsa rsa ed25519 ; do
 mv ~/.ssh/id_${type}{,.bak}
 mv ~/.ssh/id_${type}.pub{,.bak}
done

See notes** at the end to restore these original keys (after all you will need your existing keys until your CA is trusted by all endpoints).

As a sanity check, if you are using a service such as keychain or ssh-agent that remembers your keys, you might want to stop the service so you know your test is not using your old keys:

tricky@testclient $ killall ssh-agent

Create a new user keypair

You can use rsa for the type - I just prefer ed25519:

tricky@testclient $ ssh-keygen -t ed25519 -f ~/.ssh/id_ed25519
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in testkey.
Your public key has been saved in testkey.pub.
The key fingerprint is:
SHA256:0cKlJxYQrizfd5FDboyVYhn9RK7oHkpsBbarZuv+DoQ brendan@tricky-desktop.4d.eset.co.za
The key's randomart image is:
+--[ED25519 256]--+
|      ooo....    |
|     . . B.o.    |
|      + @ *o.    |
|   o o = & o.    |
|  E + . S O      |
|   + o + . o     |
|    o * + .      |
|    o= + o       |
|   ==++ .        |
+----[SHA256]-----+
tricky@testclient $

Sign the new user keypair

Keysigning makes use of serial numbers which can be used to aid in auditing. I'm not bothering with recording this in the example. You would at least increment the serial number on subsequent signing events. The serial must be a number:

tricky@testclient $ export serial=1

Identity can theoretically be anything - but client username or user@hostname is a good starting point

tricky@testclient $ export IDENTITY="${USER}@$(hostname)"

The key needs to grant access to specific remote user(s). With a typical bastion configuration, you wouldn't have 'root' here:

tricky@testclient $ export users="root,${USER}"

For a validity period, I generally recommend using 4 hours (14400 seconds) - but you may want a larger time-span. Facebook's post recommends a full day (86400 seconds)

tricky@testclient $ export VALIDITY_PERIOD=86400

Note that you sign the public key, as it is the public portion that needs to be accepted by the endpoint:

tricky@testclient $ ssh-keygen -s ${CAPATH}/ca -I ${IDENTITY} -n ${users} -V +${VALIDITY_PERIOD} -z ${serial} ~/.ssh/id_ed25519.pub
Enter passphrase:
Signed user key /home/tricky/.ssh/id_ed25519-cert.pub: id "tricky@testclient" serial 1 for root,tricky valid from 2018-10-09T20:06:00 to 2018-10-10T20:06:00

Connect to the endpoint:

tricky@testclient $ ssh root@testserver
Last login: Mon Oct  8 20:46:31 2018 from testclient.localdomain
root@testserver ~ #

* Thanks, @PatrickMevzek, for the ref info:

TrustedUserCAKeys ... was added first in a commit on March 4 2010 openssh.com/txt/release-5.4 "Add support for certificate authentication of users and hosts using a new, minimal OpenSSH certificate format (not X.509)."

**

for type in dsa rsa ed25519 ; do
 mv ~/.ssh/id_${type}{.bak,}
 mv ~/.ssh/id_${type}.pub{.bak,}
done

Once restored, I recommend signing your existing keys but also to start treating them as ephemeral. Keep a backup of your original keys in case you believe they may be needed for services that trust them but don't support CA-based Key-signing.

  • 1
    This sounds very very interesting. I'm still not sure how one gets their keys signed so quickly (you mentioned a validity of 4 hours - I can't expect an engineer to get a manual approval every day, let alone every 4 hours). The Facebook post uses a bastion to sign the keys (on login, I assume) and they protect the bastion with some heavy liftin which is completely out of scope for my 10 person operation. I guess with Vault you can automate this with an ssh client wrapper script, and then you protect access to Vault at a single location. – Guss Oct 9 '18 at 19:08
  • 1
    Netflix has an automated system that generates short term certificates (5 minutes) for its staff. See presentation at speakerdeck.com/rlewis/… and code at github.com/Netflix/bless – Patrick Mevzek Oct 9 '18 at 23:09
  • 1
    "* I'm interested in finding out how long openssh has supported key-signing :)" the option TrustedUserCAKeys that is key for the CA based authentication system was added first in a commit on March 4 2010. That would put it in Openssh version 5.5, in fact even the previous minor one if you read openssh.com/txt/release-5.4 "Add support for certificate authentication of users and hosts using a new, minimal OpenSSH certificate format (not X.509)." – Patrick Mevzek Oct 9 '18 at 23:21
  • 1
    @Guss - Ideally, it won't be manual at all, as long as the user can authenticate themselves with the system providing the signing. I use 4 hours because I feel it's sensible to do it twice per working day (once in the morning; once after lunch). YMMV of course. :) – zaTricky Oct 10 '18 at 5:51
2

I believe you are either misunderstanding or over-estimating the complexity of a bastion host.

bastion server for each product/environment ... you should only need 1x bastion host (2x for redundancy just in case).

running non-trivial installation ... if you have OpenSSH that is already a bastion host (you would just harden it as you would any other externally facing server)

I currently run a bastion host for my personal setup, here is what I do:

Bastion host - VPS (costs me $2.50 a month 1cpu 512mb 500gb/m ymmv) it has OpenSSH, UFW, Fail2Ban

Internal[0-7] - various VPS & Physical servers I manage for my friends and family ... each has OpenSSH configured to only accept connections from Bastion host

Bastion host's authorized_keys (personally moved it out of user profile so there is 1 file to rule them all and so non-admin users cant modify it)

ssh-rsa AAAA ... 4W95 myAdminUser@desktop
ssh-rsa AAAA ... rHjK alice
ssh-rsa AAAA ... DkQP bob

you could disable shell on a per key for non-admin users ... I personally trust my users to have non-root shell on the bastion host ymmv

Internal[0-7]'s /etc/ssh/sshd_config

...
AllowUsers *@<bastion.host.static.ip>
...

Desktop's ~/.ssh/config:

Host bh
  HostName bastion.host.domain.name
  ControlPath ~/.ssh/cm-%r@%h:%p
  ControlMaster auto
  ControlPersist 10m

Host intN
  HostName n.internal.domain.name
  User <username>
  IdentityFile ~/.ssh/id_rsa
  ProxyCommand ssh -W %h:%p bh

Using this ~/.ssh/config you would use the command ssh intN to connect ... alternitivly ssh user1@intN and scp intN:/foo/bar ./ also work

when someone wants access, I tell them to give me a username and a pub-key, an account is added to each server they should have access to with pub-key added to there users ~/.ssh/authorized_keys (I personally automate this w/ github's ssh-key api and Ansible ymmv)

when someone no-longer needs access I comment out # there pub-key from the authorized_keys list on that specific box

when someone no-longer needs access to anything I comment out # there pub-key from the authorized_keys list on the bastion host ... even if there key is left on the individual servers ... if they cant get into the bastion host they cant get in to anything.

  • This is an interesting approach, especially with the ProxyCommand usage. That being said, if you're setting up a user account on each internal host then I don't see the added benefit in the bastion host - it adds another layer of security, sure, but from my perspective - it isn't worth the extra complexity. Using a bastion make sense if the per-user key is terminated at the bastion and internals don't have to manage a dynamic set of keys (I like my servers immutable), but then its hard (impossible?) to prevent a user from stealing the bastion's key. – Guss Apr 11 '18 at 18:25
  • I do want to commend your solution though - it has the sanest revocation process I've seen so far, its only downside is that to keep things simple (and immutable) I'd skip revoking on the internals and just revoke on the bastion, which means that internals will collect garbage over time. But with auto-scaling servers its not a big issue. – Guss Apr 11 '18 at 18:33
  • @Guss in terms of the annoyance of creating a new account for each host ... I am 100% sure that can be automated (through FreeIPA ... as that is how my company does it) ... though I cant speak to the complexity or configuration of that aspect as I didnt set it up. At work I simply ssh to a new box (through the bastion host) and it builds a new home dir and all my permissions work. – CaffeineAddiction Apr 12 '18 at 4:39
  • The LDAP approach (or FreeIPA which is basically LDAP with a bunch of bells and whistles on top) was suggested before. I really believe that adding LDAP centralized authentication to my auto-deployed and auto-scaled application stack (which I currently have ~10 different copies of with different users) is going to be a maintenance nightmare. Building it once is hard, building it to auto-deploy in a larger stack is very hard, maintaining such a setup on 10 different installations, that keep changing, is not something any system admin will be happy about. – Guss Apr 12 '18 at 11:35
1

Maybe a solution such as FreeIPA could help you. It provides a way to make sure that ssh keys are properly distributed as well as sudo profiles and authentication.

Z.

  • 1
    An interesting concept, but I don't believe this is the right solution for me: FreeIPA is kind of heavy lifting and requires installing software on each server (what FreeIPA calls "client") and its use case (SSO to per-user named accounts) doesn't really fit the EC2 model (the only account on the server is the one with sudo permissions and the public part of the AWS key pair). – Guss May 4 '15 at 16:50
  • You don't actually need the client, you could pick and choose only the pieces you want, and manually manage the configurations. Like Linux to AD style, the FreeIPA client packages are just scripts that automate the various configuration changes. Their are no agents or anything like that, unless you count SSSD as one.. but again, that's a choice. – J. M. Becker Nov 7 '16 at 17:46
1

A typical best-practices answer would be to use something like LDAP. With it, you can define users, groups, and more.

By connecting your SSH authentication backend via PAM to your LDAP server, you can maintain separate accounts for each member of your team. I may be mistaken, but you should also be able to maintain separate $HOME/.ssh/authorized_keys files for each user as well.

By making users members of the sudoers group, they'll be able to request root permissions on-demand. These accesses will be recorded in logs, so you'll have auditability later on down the line.

This setup is pretty standard in larger corporations.

  • 1
    You're not the first to suggest that the best way to manage security in a dynamic cluster to individually manage user credentials on all servers. I don't think this scales - even if I could deploy LDAP (which I can't due to network partitioning). How would you get the keys of a new member to all servers? For the sake of the argument, here's how yesterday went - I deployed two new clusters (>5 servers each) and recycles 8 servers in another network (i.e. killed them and created new ones). If I spend only 5 extra minutes on each server setting up security, the system will never be ready on time. – Guss Aug 26 '15 at 8:54
1

When I need to distribute public ssh-keys on my servers, I use an online tool (https://keydistributor.io) and fetch the keys on each server from their API. In that tool (just like Puppet, chef etc) I could define groups, where each user belongs to a group.

  • This looks like an interesting service. It's a bit expensive when you have many servers and I'm not sure how they handle auto scaling servers - but it's a step in the right direction. I'm not sure why some one word choose to down vote you. – Guss Nov 28 '16 at 17:40
  • @Guss look at Azure Keyvault if you're looking for something scaleable – Little Code Nov 28 '16 at 18:12
1

A simpler way is as shown below.

For Linux / Mac users:

To create Public and Private keys use the following command: ssh-keygen -t rsa -b 4096

Upload the public key to a folder in your S3 bucket. For example: S3 > MyBucket > Keypair

Save and secure your private key.

For Windows users:

Use puttygen to generate the keys. Upload the public key to S3 > MyBucket > Keypair

Save and secure your private key. The following steps are important during the launch of any Linux AMI.

Ensure the IAM role has a role created with AmazonS3FullAccess policy. This allows the instance to assume a role to access the S3 buckets. This is needed to read the public keys from S3 and copy them to the user profile.

Add the following code under the user-data section in Configure Instance details > Advanced Details (as Text):

    #!/bin/bash
    usermod user 1
    usermod -aG wheel user1
    mkdir /home/user1/.ssh/
    aws s3 cp s3://MyBucket    /Keypair/user1-pub.pub /home/user1/.ssh/authorized_keys

    useradd user2
    usermod -aG wheel user2
    mkdir /home/user2/.ssh/
    aws s3 cp s3://MyBucket   /Keypair/user2-pub.pub /home/user2/.ssh/authorized_keys

    sudo -i 
    echo “user1 ALL=(ALL) NOPASSWD:ALL” >> /etc/sudoers
    echo “user2 ALL=(ALL) NOPASSWD:ALL” >> /etc/sudoers

yum update -y

This setup creates User1 and User2 and adds them to sudo users. The aws s3 cp command copies the users public keys from the S3 folder to their .ssh/authorized_keys path. The last section is to run commands as admin without needing passwords.

There are lots of security improvements that can be recommended here. While not explicitly used in this example, limiting S3 bucket access to a specific bucket and knowing the security implications of disabling password usage in sudo, are few things that can be highlighted. Use them wisely based on your particular needs.

  • This is a relatively simple and sane approach, kudos. You might want to consider moving to AWS Secrets instead of plain S3 as it is more geared towards this scenario. My main problem with this approach is that it again forces me to setup each internal servers with multiple user accounts. With dozens of internal servers, some of them auto-scaled and going up and down, some persistent for a long time, managing revocation is a nightmare. – Guss Apr 11 '18 at 18:30
0

Opera Software's SKA - SSH Key Authority offers an interesting solution: as far as I understand it is a central server that allows you to register servers and setup rules for how users (either manually managed or synced from LDAP) have access or admin rights in registered servers, then users can upload their public keys and SKA publishes keys to servers by SSHing in as a pre-prepared "sync-keys" users and updating a set of SKA specific files that sshd has to be configured to read.

On the face of it I don't like it because:

  • servers have to be registered ahead of time. Setting up automatic registration (and worse - automatic deregisteration) will be quite a hassle.
    • I'm not happy about the requirement to configure all servers with a specific user and configure sshd in a non default way, that will break access for the cloud platform's default user account (so if the configuration breaks, there's no way to fix it or figure out why it broke).

Otherwise, it looks very promising, especially the semi-automatic team management and logging & notifications.

0

If you don't want a key distribution machinery then using a SSH-CA for issuing tempory OpenSSH certificates might be the right choice.

With this you only have to install the SSH-CA's public key as trust anchor on the target machines (see sshd_config directive TrustedUserCAKeys).

Caveat: This currently only works with OpenSSH and libssh.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.