Dealing with an insider threat in an insecure environment

Question

I'm not a security expert by training but I know enough to deal with the occasional breach and how to set up basic practices around SSH keys, passwords, and configuration management in general. I recently had to clean up a bitcoin miner from one of our hosts. The way I discovered it was that the miner was hogging all the CPU and running strings on the binary showed upx packer and other malicious looking strings. The whole thing smelled fishy from the start in terms of how the miner got there. I have very strong suspicion it was an inside job.

Now, the security profile at this place is very lackadaisical and I'd like to lock things down without tipping off whoever installed the miner because the obvious follow up is to fire the person and I want to mitigate the inevitable damage that someone with their current level of access can do. I doubt they're competent enough to do further damage because they already played their trump card and it was dealt with so they're less likely to pull the same thing but they might get other ideas and sneak in less visible exploits onto the servers.

What is the standard protocol for dealing with such situations and what is the checklist I should go through to weed out all potential insecure access paths without tipping off the insider?

There is no IT or security team. I'm it as far as security teams are concerned. I'm not looking for political cover games. I need a list of technical things I need to do to lock things down. I'm not looking for answers about politics and chains of command.

Answer 1

I am not a security expert, my experience is limited to admin on a small unix server, so I only know the basics. But I am going to answer your question because the solutions to your problem involve too many (organizational/political) pitfalls for a professional to venture into, as you have already seen.

The crux of your current problem is that someone installed unwanted software on your systems. The solution to that problem does not require a "lock-down", and a lockdown will be overkill - you will lose credibility, and the insider will be tipped off. You lock things down when you are under attack. A simplistic example of an attack is a ddos. So as a first step, sit back and see things in perspective - what has happened here is not too different from someone installing a personal mail client for legit purposes on company hardware.

In general, the greater the "security", the greater the hassles for day-to-day work. For a startup, overdoing the security bit will demotivate people and ruin the culture. There are two parties at fault in your case - 1) the person who installed the miner, and 2) the person who set up the systems in such a way that anyone could install software. The second more than the first.

In a startup, people have a lot more leeway than in large companies. Part of that leeway is allowing people to install the programs they need without having to go through a (long and often frustrating) chain of command. At 10 people, this arrangement is already starting to break.

As you mention, the security culture in the organization is lackadaisical. Make sure management understands this, use this case as an example. Once they do, get one of them (management/founder/etc.) to institute a policy about the company needing proper security practices, and send out a memo to that effect, so the whole thing looks routine.

Start reading standard security practices and follow most of them. For a good reference, get the book "Absolute FreeBSD - the complete guide to freebsd" and completely read the chapter titled "Securing your system". It covers the typical threats, as well as standard practices. But really, read up standard security practices - there are lots of good resources online, find them, study them carefully, and follow them. Or hire someone who can/already has. Lastly, understand that security is always a tradeoff.

Some simple examples of these standard practices are -

1. User account security - only very few users have sudo rights on production servers, or company-wide resources. People can admin their own local machines, but only very very few can admin devices across the whole network.
2. Groups - create appropriate groups with the right set of permissions, and assign user accounts to these groups. So only certain groups have access to specific resources. This will allow you to avoid giving root access to too many people. You need to think how this applies to your particular setup.
3. Network security - disallow password based remote logins, stick to ssh only. Very very few should be able to ssh into the prod server. Have good firewalls, the usual.
4. Make sure all local login passwords are strong and changed frequently.

Read the discussion on this security.se question and the ones linked therein.

TL;DR - this is not a disaster, so chill, sit back and think rationally. There's no need for a "cleanup" or a "lockdown". Use this incident as a case-study to convince management to adopt a policy about following standard security practices. Learn about standard security practices and follow them. Understand that security is a tradeoff, you have to make the choices appropriate to your startup.

Answer 2

If IT security is not your job, stop right now. Report on the official channels and move on. This is the standard procedure. Messing with the miner or doing your own investigation may create evidence against you.

Report and have IT lock down the affected machine to start an official investigation. Or you risk being accused by the attacker of installing the miner yourself and trying to frame him.

If you company is a small startup, you need permission from someone. Call the owners or the ones in charge, show them what you found and ask permission to investigate.

Shutdown the server in front of them, boot a live distro, and take a image of the disk. Hash it, and send them the hash (to prove you haven't tampered with it) along with the image size, and a list of files on the disk before you start working. Mount the image read only on your computer, and create a detailed report.

With this, you have witness backed evidence.

Answer 3

It depends on your role at your current organization. If your organization already has an existing IT-security team, report it to them right away.

If you're a small enough organization where you are the IT-security team, then you're going about things the right way -- however, you still need to immediately report your findings to your superior.