When a sysadmin leaves what extra precautions need to be taken?

Question

If a user leaves an organisation who knew most of the top credentials are there any other precautions that need to be taken other than changing those credentials?

Obviously there is also the standard user leaving stuff like email/VPN access being revoked, mac addresses of their devices being removed from our network etc etc. but I am more worried about power or super users.

This question was IT Security Question of the Week.
Read the Aug 19, 2011 blog entry for more details or submit your own Question of the Week.

Answer 1

An admin should never have "super user credentials" that cannot be removed by simply removing his user account or move the user account to a group that lacks the permission.

An example: The admin logs in to a Linux system with his own account and uses sudo somecommand to do things that requires root permission. You don't allow this admin user to actually log in as root (and thus knowing a single root password that would have to be changed). There IS NO ROOT ACCOUNT to login to.

I think this principle can be applied to other access credentials as well.

Of course this only works if everyone actually follows the policy before they quit their job or get fired (or are moved to another part of the company). It does not protect against an admin who willingly breaks the policy and creates a back door for himself.

Answer 2

It does depend quite a lot on if the admin left on good terms and how complex your network is but some good steps to take are. I have worked with a number of organisations that take some of these steps but none of them take them all. A lot of them only revoke the user credentials and only take further steps if the admin left on bad terms or was going to a competitor.

• Revoke the users credentials across all servers network wide. This should include keys as well as passwords.
• If they are aware of root details that do not require being logged in as another user first then these details should be changed as well.
• Other account details such as those relating to server hosting should also be changed as well, as there is always a chance they have a copy of them or can remember them.
• If your network is closed of but you allow traffic from certain IP address such as admin home address in it is important to ensure that those are removed from access as well.
• If possible monitor for attempted logins by the leavers username this would give you an alert as the potential that they are trying to gain access to your systems.

Answer 3

Good luck with that. Normally you make sure that all of these things have been done before you lay somebody off. If someone is a bit skilled it can get very hard, if not almost impossible to see if he hasn't compromised a system.

If he was the only employee, there is no saying if he has any backdoors in place. So at this moment he still might have access. When you give a system administrator access, you gain his trust and you have to make sure that he's trustworthy as well.

So things to check:

• monitor all incoming and outgoing traffic

• revoke all of his access (should be done before you lay someone off)

• Documenting

From my personal view:

You seem to have no experience with this and it can be quite dangerous, you should get a professional consultant. I would at least have talked to the guy and asked what he has done. Remember he had access to everything, he can break anything. He knows his infrastructure. If he does something malicious, you can start from scratch because none of your current systems can be trusted.

Should you not believe me, have a look at these cases:

• DHL disgrunteled employee deletes all orders of 48 hours
• Chevron — Emergency system was sabotaged by disgruntled employee in over 22 states (USA 1992)
• Terry Childs, who shut down FiberWAN

As @Joel stated: People think Jurassic Park is about dinosaurs eating people and shit, but it's really about what happens when you don't compensate your sysadmin appropriately

update

The company should have told him they were going to hire somebody in a fixed position a little bit more upfront. Then explain him that he could still help from time to time if something went wrong. Also you should have a longer transition period. If he's not cooperating you basically have the Terry Childs scenario. So apart from prosecuting him, there is little you can do.

Answer 4

For the benefit of everyone else who will be reading this, this is definitely the wrong time to be thinking about this.

Now, you've put yourself into a literally impossible position; you simply cannot tell with certainty that your system is not compromised. Furthermore, the more difficult your sysadmin was or how difficult a time you may have given him, the more likely it is that he will have installed some sort of back door. As someone who cleans up this sort of mess for a living, I can tell you that the percentage of sysadmins who do this is disturbingly high, and the danger is very real.

On the subject of what to do now, the primary determining factor is how much downtime you can tolerate and how critical your IT system is to your business. Ideally, you shut everything down, and rebuild it all from scratch. Yes, really. This sounds extreme, but now that you've gotten yourself into this mess, it's the only way to be sure. No other solution will work.

If you can't afford that, or if you think you can tolerate a serious compromise, you can instead comb through the systems one-by-one looking for any long-running processes, scheduled tasks, hard-coded authentication tokens, authentication back-door code, remote management processes, VPNs, keystroke loggers or other monitoring tools, any other components inconsistent with your security policy.

And finally, if you can't afford that, then you can always close your eyes, cover your head, and hope for the best.

How to prevent this in the future

More importantly, here's a few tips on keeping this from happening again:

• Have multiple top-level sysadmins. Hierarchies are easy for management, but bad for security. You need someone who can "watch the watcher"; someone who has the skill and the authority to check up on what your sysadmin is doing and who can find these backdoors. A sysadmin is far less likely to put in a backdoor if he's afraid that someone will find it.

• Prefer honesty over skill when you hire a sysadmin. You need someone you can trust implicitly, someone you know isn't going to hang you out to dry. Even if they aren't the most skilled administrator, their primary role is to be responsible for your defenses -- honesty matters more than all other qualifications combined.

• Don't tolerate condescending superiority. This goes along with the point above, but to add a bit of clarity: someone who doesn't respect others cannot be trusted to act in best the interest of others. Such a person should not be allowed to work in IT in your company.

• Keep your sysadmins happy, whatever that might mean to them. If they aren't happy, then either fix the problem or find someone else -- and fast.

Answer 5

who knew most of the top credentials

I am not exactly certain of your meaning, but this scares me. It sounds like the individual had extensive access and control over many critical systems. The sysadmin may even have had exclusive control. A few years ago the city of San Francisco lost control of it's network because a single sysadmin refused to give his passwords to his superiors. So, way before a sysadmin leaves, you should make sure that there are at least two people who have access and control over critical systems.

are there any other precautions that need to be taken other than changing those credentials?

Revoke physical access.

It's probably standard to collect badge and office keys, but don't forget keys to cabinets, restricted areas, alarm codes, off-site storage locations, parking permits, etc. You should be keeping a record of every physical key given out, who received them, and why. Additionally you should be able to rekey critical access points within 24 hours. If your alarm system allows for a user to call in a false alarm, make sure the leaving employee is removed from that list.

Let everyone know.

It's not hard for a former sysadmin to social engineer a current employee. It's virtually impossible to prevent them from calling a current employee. Given the access to internal information the sysadmin likely knows who to call and what to say to gain some type of network access. So, announce that the individual is no longer an employee. If possible show a picture (a badge photo does nicely) of the ex-employee. Make sure they know who to speak to if the former sysadmin makes suspicious contact. If the sysadmin had a close relationship with any suppliers, partner companies, or subsidiaries, let them know as well.

Don't forget the phone and voicemail system.

The vulnerabilities here range from snooping on random voicemail to making long distance calls at the company's expense.

Credit cards, charge accounts, Purchase ordering

If the sysadmin had authority to order equipment, make sure that ability is revoked from all suppliers, venders, etc. Its trivial to order some nice equipment using the standard procedure and change the shipping address.

Review any contracts and agreements signed by the former employee.

If the sysadmin signed any contracts or agreements as part of their job, have someone review all documents that they signed. Ideally you should have a database of documents that employees have signed and be able to instantly call up every document that a given employee signed.

Check the patch and update status of your critical systems.

Even the the former employee did not act as administrator for those systems, they may know what their status was.

Answer 6

After reading ALL that + commentary...

You're screwed. Start from scratch. Harden systems, new keys/passwords, VLAN's, VPN...

Basically, the whole nine yards.

Come @ it from the perspective of a VERY disgruntled ex-admin. Then, defend against those conceivable vectors. He worked remotely, restrict all other remote access except for your VPN(with refreshed keys).

Its a basic batten down the hatches routine for which most companies have policies & procedures written well in advance.

Hope you make it through, but, if you came here for that much information about what to do, I feel sorry for the company. :-(

Answer 7

I had the unfortunate experience of almost the same issue. I lost several nights of sleep over this, I hope this helps you.

Short Answer: Start from the outside in. If your applications/internal operations would allow changing IP addresses (if you know they will attack) could be a game changer. In my case I could not do this since the applications ran from the remote sites were directly hitting an external IP.

Second I disabled the office's WiFi that he(previous admin) setup. I did this second since I knew I would be able to see him on the property if he were to try to access. Also look to see what wireless signals you see walking around the premesis to try to eliminate the possibility of a hidden access point installed without anyones knowledge.

The next thing I did was audit EVERY RULE in my firewall. I found a few static IP's (aircard and cable internet) that were "allowed" and just denied access from everywhere that I could not verify that needed access.

Next I went through every log file on every server to look for failed login attempts. If you find anything document it.

Then I would force ALL USERS to change their passwords and enforce a policy that does not allow words from a dictionary as a password. I had to convince management of this but when I showed an attack using a dictionary file VS bruteforce and they approved the company change that minute.

Next (atleast in my case) was the phone system. Change all access codes for everyone and force them to use somethign else (or do it for them if they refuse to change their voicemail PIN)

The second to last thing to do would be to sniff all of your own traffic. Buy or build an inexpensive LAN Tap (I bought a throwing star lan tap) for about $20 and I put it in between the firewall and the main switch that everything comes into. I paid extra close attention to traffic outside of business hours. This should help you identify what goes on, on your network during non-business hours.

Lastly, depending on the situation as an entirety (did your employer fire them to save money?) you may have to give them (previous advin) a call depending on the complexity of the situation. I found out who was friends with the previous admin and became friends with them quickly, in case I needed to call the old admin for a favor (password to web hosting of a seperate domain that the company purchased as a gift for a non-profit group) The non-profit company that the old admin wrote a site for was still lost since they didn't pay him (previous admin) for another year to renew the domain but atleast I got the domain back for the non-profit.

If the previous admin does make any verbal or even (dumber) written threats keep multiple copies of everything. You can also state that the legal issues that could come of it aren't worth the effort (which they never are). Hopefully this will help you a bit.

Answer 8

NOTE: this answer was migrated from another question and then merged into this one. Therefore it does not quite apply to this question (the other question was describing a situation where many mistakes were already made and access was still available to a soon-to-leave sysadmin). Perhaps it should be removed, but I left an unedited version below

---

I think the other answers (particularly @tylerl, @lucaskauffman) are absolutely right, but at the same time perhaps they are over-alarming.

Yes, you cannot know for sure what this disgruntled sysadmin might do. However, it takes quite some discontent to make someone break into the system and cause damage. This kind of action is most probably both in breach of their working contract and illegal (as in you can go to jail if you are caught).

From the way I look at it, the crucial point other answers miss, or treat as an afterthought, is how to fix the root cause or your highest risk, which is how upset this person might be.

I therefore think the first and best course of action, is to try to make this person the least discontent, possibly even happy. Offer them extra money for handing things over, documenting what they were doing and making sure the successor has an easy job. Offer even extra 'standby' compensation when they get paid but don't have to do any work. Explain why for the company it makes sense to have an internal employee doing this work rather than an external contractor.

At the same time make it clear that you know they are the only people capable and having full access to the system, and if there are any breaches, they are going to be the prime suspect and that you won't hesitate to report and investigate even the smallest incident.

Bringing your entire IT down and building it from scratch will fix it too, but I don't see how realistic this solution really is.