Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.
2 deleted 5 characters in body
source | link

The key is to only implement two-man control for operations that really need it.

Routine tasks

Routine tasks, like deploying an application release, or applying security patches can be scripted. One sys-admin can create the script, but it cannot be run until a second sys-admin has reviewed and approved it. Common tasks like restarting the application server, or deploying the latest application release, can have a pre-approved script that is used repeatedly. Non-standard tasks will need a bespoke script.

To do this, there's a need for some kind of workflow management system. You may be able to use a ticketing system like Trac, build a bespoke system on something like JIRA, and configuration management tools like Puppet may have features that would help. The two sys-admins do NOT need to be physically local; they can work remotely.

The systems should be designed so that as much routine maintenance is automated as possible - scheduled log rotation, etc.

Diagnostics

Sys-admins can have read-only access to logs without two-man control, provided the logs mask sensitive information. They can also have a low-privilege account to view status pages, and run commands like top or ps. This allows a single sys-admin to diagnose issues - and they can work remotely. They may be able to do remediation by submitting a script as a routine task. However, in some cases that won't be possible, so there is a need for emergency access.

Emergency access

Sometimes there is a serious issue and you just need to SSH in as root. To maintain two-man control you need two people at this point. In practice, they need to be physically next to each other. A simple way to implement access control is to have an account for each pairing, where each sys-admin knows half the password. You could create a smarter system with a custom PAM module, perhaps even requiring a separate smart card from both sys-admins.

Once logged in, it is down to the sys-admins to maintain two-man control. They must inspect each others commands before issuing, and lock the terminal if one of them needs to step out.

The aim should be that emergency access is rarely needed.

Protect live onlyDevelopment environments

It is only necessary to use two-man control for the live environment. A lone sys-admin can have full access to a development environment - provided that live data is not present. A lone sys-adminThey can try out a task in the development environment, perform some testing, iron out the bugs. Only when they have got a working script do they submit it to another sys-admin for approval.

Conclusion

If all this is done, I believe you can implement strong two-man control with about 25% overhead. Making it a success relies on various things (like automated maintenance) that are good practice anyway. If a system is so sensitive as to need two-man control, it is a good idea to get all the basics right as well!

I have never seen this implemented in a commercial or government environment. The closest I have seen was an online banking platform where most sys-admins did not routinely have root access. To performs a task, this would first be planned in detail, then approved on the change control system. To implement the change the sys-admin then got root access for a limited period. Also, there were a small number of senior sys-admins who had permanent root access, in case of emergencies. This arrangement is much better than I typically see - although it falls short of full two-man control.

The key is to only implement two-man control for operations that really need it.

Routine tasks

Routine tasks, like deploying an application release, or applying security patches can be scripted. One sys-admin can create the script, but it cannot be run until a second sys-admin has reviewed and approved it. Common tasks like restarting the application server, or deploying the latest application release, can have a pre-approved script that is used repeatedly. Non-standard tasks will need a bespoke script.

To do this, there's a need for some kind of workflow management system. You may be able to use a ticketing system like Trac, build a bespoke system on something like JIRA, and configuration management tools like Puppet may have features that would help. The two sys-admins do NOT need to be physically local; they can work remotely.

The systems should be designed so that as much routine maintenance is automated as possible - scheduled log rotation, etc.

Diagnostics

Sys-admins can have read-only access to logs without two-man control, provided the logs mask sensitive information. They can also have a low-privilege account to view status pages, and run commands like top or ps. This allows a single sys-admin to diagnose issues - and they can work remotely. They may be able to do remediation by submitting a script as a routine task. However, in some cases that won't be possible, so there is a need for emergency access.

Emergency access

Sometimes there is a serious issue and you just need to SSH in as root. To maintain two-man control you need two people at this point. In practice, they need to be physically next to each other. A simple way to implement access control is to have an account for each pairing, where each sys-admin knows half the password. You could create a smarter system with a custom PAM module, perhaps even requiring a separate smart card from both sys-admins.

Once logged in, it is down to the sys-admins to maintain two-man control. They must inspect each others commands before issuing, and lock the terminal if one of them needs to step out.

The aim should be that emergency access is rarely needed.

Protect live only

It is only necessary to use two-man control for the live environment. A lone sys-admin can have full access to a development environment - provided that live data is not present. A lone sys-admin can try out a task in the development environment, perform some testing, iron out the bugs. Only when they have got a working script do they submit it to another sys-admin for approval.

Conclusion

If all this is done, I believe you can implement strong two-man control with about 25% overhead. Making it a success relies on various things (like automated maintenance) that are good practice anyway. If a system is so sensitive as to need two-man control, it is a good idea to get all the basics right as well!

I have never seen this implemented in a commercial or government environment. The closest I have seen was an online banking platform where most sys-admins did not routinely have root access. To performs a task, this would first be planned in detail, then approved on the change control system. To implement the change the sys-admin then got root access for a limited period. Also, there were a small number of senior sys-admins who had permanent root access, in case of emergencies. This arrangement is much better than I typically see - although it falls short of full two-man control.

The key is to only implement two-man control for operations that really need it.

Routine tasks

Routine tasks, like deploying an application release, or applying security patches can be scripted. One sys-admin can create the script, but it cannot be run until a second sys-admin has reviewed and approved it. Common tasks like restarting the application server, or deploying the latest application release, can have a pre-approved script that is used repeatedly. Non-standard tasks will need a bespoke script.

To do this, there's a need for some kind of workflow management system. You may be able to use a ticketing system like Trac, build a bespoke system on something like JIRA, and configuration management tools like Puppet may have features that would help. The two sys-admins do NOT need to be physically local; they can work remotely.

The systems should be designed so that as much routine maintenance is automated as possible - scheduled log rotation, etc.

Diagnostics

Sys-admins can have read-only access to logs without two-man control, provided the logs mask sensitive information. They can also have a low-privilege account to view status pages, and run commands like top or ps. This allows a single sys-admin to diagnose issues - and they can work remotely. They may be able to do remediation by submitting a script as a routine task. However, in some cases that won't be possible, so there is a need for emergency access.

Emergency access

Sometimes there is a serious issue and you just need to SSH in as root. To maintain two-man control you need two people at this point. In practice, they need to be physically next to each other. A simple way to implement access control is to have an account for each pairing, where each sys-admin knows half the password. You could create a smarter system with a custom PAM module, perhaps even requiring a separate smart card from both sys-admins.

Once logged in, it is down to the sys-admins to maintain two-man control. They must inspect each others commands before issuing, and lock the terminal if one of them needs to step out.

The aim should be that emergency access is rarely needed.

Development environments

It is only necessary to use two-man control for the live environment. A lone sys-admin can have full access to a development environment - provided that live data is not present. They can try out a task in the development environment, perform some testing, iron out the bugs. Only when they have got a working script do they submit it to another sys-admin for approval.

Conclusion

If all this is done, I believe you can implement strong two-man control with about 25% overhead. Making it a success relies on various things (like automated maintenance) that are good practice anyway. If a system is so sensitive as to need two-man control, it is a good idea to get all the basics right as well!

I have never seen this implemented in a commercial or government environment. The closest I have seen was an online banking platform where most sys-admins did not routinely have root access. To performs a task, this would first be planned in detail, then approved on the change control system. To implement the change the sys-admin then got root access for a limited period. Also, there were a small number of senior sys-admins who had permanent root access, in case of emergencies. This arrangement is much better than I typically see - although it falls short of full two-man control.

1
source | link

The key is to only implement two-man control for operations that really need it.

Routine tasks

Routine tasks, like deploying an application release, or applying security patches can be scripted. One sys-admin can create the script, but it cannot be run until a second sys-admin has reviewed and approved it. Common tasks like restarting the application server, or deploying the latest application release, can have a pre-approved script that is used repeatedly. Non-standard tasks will need a bespoke script.

To do this, there's a need for some kind of workflow management system. You may be able to use a ticketing system like Trac, build a bespoke system on something like JIRA, and configuration management tools like Puppet may have features that would help. The two sys-admins do NOT need to be physically local; they can work remotely.

The systems should be designed so that as much routine maintenance is automated as possible - scheduled log rotation, etc.

Diagnostics

Sys-admins can have read-only access to logs without two-man control, provided the logs mask sensitive information. They can also have a low-privilege account to view status pages, and run commands like top or ps. This allows a single sys-admin to diagnose issues - and they can work remotely. They may be able to do remediation by submitting a script as a routine task. However, in some cases that won't be possible, so there is a need for emergency access.

Emergency access

Sometimes there is a serious issue and you just need to SSH in as root. To maintain two-man control you need two people at this point. In practice, they need to be physically next to each other. A simple way to implement access control is to have an account for each pairing, where each sys-admin knows half the password. You could create a smarter system with a custom PAM module, perhaps even requiring a separate smart card from both sys-admins.

Once logged in, it is down to the sys-admins to maintain two-man control. They must inspect each others commands before issuing, and lock the terminal if one of them needs to step out.

The aim should be that emergency access is rarely needed.

Protect live only

It is only necessary to use two-man control for the live environment. A lone sys-admin can have full access to a development environment - provided that live data is not present. A lone sys-admin can try out a task in the development environment, perform some testing, iron out the bugs. Only when they have got a working script do they submit it to another sys-admin for approval.

Conclusion

If all this is done, I believe you can implement strong two-man control with about 25% overhead. Making it a success relies on various things (like automated maintenance) that are good practice anyway. If a system is so sensitive as to need two-man control, it is a good idea to get all the basics right as well!

I have never seen this implemented in a commercial or government environment. The closest I have seen was an online banking platform where most sys-admins did not routinely have root access. To performs a task, this would first be planned in detail, then approved on the change control system. To implement the change the sys-admin then got root access for a limited period. Also, there were a small number of senior sys-admins who had permanent root access, in case of emergencies. This arrangement is much better than I typically see - although it falls short of full two-man control.