How to secure shared user on build server?

Question

We have a CI server which is installed as the builduser and all tasks are executed as that respective user at the OS level. We have many teams sharing the build server which means the build user and its resources are shared.

Each team also manages different environments and this requires ssh connectivity from the build server. Because we are driving automation, passwordless connectivity was initially setup to facilitate running automated tasks remotely (build user's public key copied to the deploy user's authorized_keys file on the remote host).

This as you can imagine poses a security concern since any user running a task can access any environment.

One solution would be to use something like sshpass (provide the ssh password as an argument) and have different user/password combinations for each environment. At the task level create role based ACLs to lock down who can see the password as it seems like it will be in plaintext. Another concerning thing I'm curious to find out more about is the command exposure here. I have been lead to believe that users can peak into the running process and see what commands the build user has executed - is this possible without root privileges? (even better if someone can illustrate how this can be done or point me to some documentation)

Apart from validating my approach, I'm also looking for recommendations on how this problem can be potentially solved.

Answer 1

If the build account can execute arbitrary commands as a production user on a production server, it will need to be locked down. Only a select group of privileged users should be able to use it, ideally without the ability to alter the (hopefully small set of pre-defined) commands.

Do you really need this ability? Maybe there's a better solution, such as a drop area on the production server(s) that the build account can write to, complete with scripts on the production server(s) that quickly move the payload into the appropriate area (so it doesn't get overwritten), extract it, and then install it as needed. The installation would be done with a application-specific user account rather than a generic build or deploy account.

When I have set this kind of thing up in the past, I made copious use of sudo and /etc/sudoers to limit individual users' ability to act as any role account (such as build). I also used drop areas, which were typically either SFTP or revision control (subversion, git, etc).

Revision control is nice because the logs are more accessible than sudo logs. Just add a tag and the cron job that runs every five minutes will recognize the tag, check out your update, and kick off a new build. Successful completion of the build (and a smoke test suite) automatically triggers installation on a staging server, after which the QA team gets an email. When QA is content, they add another repository tag and the staging package is installed in production.

Answer 2

Figured I'll answer this with my findings since I have not received a response.

have been lead to believe that users can peak into the running process and see what commands the build user has executed - is this possible without root privileges? (even better if someone can illustrate how this can be done or point me to some documentation)

This is actually possible with very little effort. a ps -ef | grep build will show you commands run as the build user along with arguments. This means passwords can be exposed to a different user with malicious intent. Even environment variables can be viewed via /etc/proc.

The more I read about it the more I'm led to believe that it all comes down to 'trust'. A CI server is generally setup for small teams. To foster collaboration a certain amount of trust is bestowed among team members.

In my use case above, it is essentially setup as a general purpose CI server. Here we have multiple teams sharing it to run tasks. I'd say this breaks the 'trust model' unless some segregation can be provided.

Long story short, We have decided that the CI server will be under Operational control for any changes. This is admittedly a temporary solution to support a shared model of working.

In the future we will provision remote agents and associate them per team.