In this Facebook article, how do they prevent the engineers from copying SSH certificates from bastion hosts to personal laptops?

Question

These days I'm reading this Facebook engineering article Scalable and secure access with SSH. Most of the article makes sense to me but I am not experienced enough to understand the paragraph about bastion hosts:

Security domains

... In our infrastructure, engineers don't have direct SSH access to production systems, enforced by network firewalls. Instead, we use a bastion host to reach production. Engineers can authenticate to those bastion hosts only from trusted networks. These hosts use centralized LDAP and Kerberos installations to share account information, and they require two-factor authentication to protect against password leakage. Once users are properly authenticated and authorized, a background process contacts our internal CA to request a signed SSH certificate. That certificate contains all principals allowed for that specific engineer. From the bastion host, having obtained the appropriate certificate, engineers can SSH into production machines as either root or other lower privileged users. This way, we make sure that no engineer has more access than he or she requires.

We do not place SSH certificates on laptops because it is difficult to control everything that individual employees run on them. Even though they are centrally managed, laptops are more prone to vulnerabilities than the bastion servers. Therefore, we do not trust them with SSH private keys.

Unfortunately, the article doesn't explain further how the engineers log in the bastion hosts and where the SSH keys are stored. According to the paragraph, it seems to me that the engineers log in a bastion host first (from their own laptops), then after the certificate is obtained and put on the bastion host, the engineers will be able to log into the production machines from the bastion host.

The second paragraph says they "do not place SSH certificates on laptops" which suggests that they seem to have a way to prevent the SSH certificates from being copied to engineers' personal laptops. However, being a security novice myself, I have hard time imagining how they can prevent this from happening. Particularly, if the engineers log in the bastion hosts via SSH, it looks like a simply scp or rsync can copy the SSH certificates back to the personal laptops.

Any ideas how it can be implemented?

My other ideas are:

• Maybe the engineers do not log into the bastion hosts via SSH but via some other protocols that can prevent such copy operations. Is this possible?
• Maybe the bastion hosts have some special firewall rules that prevent such copy attempts?

Answer 1

Often, when an admin wants SSH access to a server, they create a client cert, store it on the device (e.g. the admin/engineer's laptop) and access the server using a direct network connection from the laptop to the server and using the cert as authentication.

Facebook is saying that they have gone a step further and added PAM (Privileged Access Management) by cycling the certs and a bastion host to prevent a direct connection to the servers from arbitrary connections.

The only way to reach the production server is from the bastion hosts. So, putting a cert on the laptop would do nothing since the laptop cannot connect to the server.

They also generate new certs for the user when they connect to the bastion host. That way the admin is always using new certs and any old certs that may have been compromised are dead, so even if the cert is copied out, it would be useless.

So, all this has nothing to do with preventing the copying of certs. The article is explaining why they use a bastion host process and how they manage their certs.

Note that all of this is in the article you linked.