2

I had a VM based on Ubuntu 16.04.03 and the only package added to it was OpenVPN. The purpose of it was to provide a VPN into my private network sited within the cloud environment. The hypervisor is also 16.04.03.

The other day I noticed 100% CPU utilization on it and once I logged in I noticed a "yam" process consuming all the user cycles. Google search revealed it likely to be a bitcoin miner.

None of the other VMs on the system or the hypervisor seemed to be affected. I have 20 hyperthreaded cores on that system so I reasoned that if the malware had been capable of it they would have gone after 40 threads instead of the 2 they got.

So I killed the process then took down the VM with the intent of rebuilding it in the morning. Foolishly I didn't take other actions.

In the morning I discovered that all the VMs on that system were down and the passwords had been changed on the hypervisor. I could log in via an SSH authorized_key authentication but could not sudo with my regular password. Worse, the ZFS zpool had been destroyed. All the Ubuntu updates got applied.

So I had to go to the co-hosting location site (fortunately local) and recover the system with console access. I was able to bring up the storage array and the other VMs. I changed all the passwords (quite a lot of them quite a pain.)

So my questions are as follows:

  1. What exploit might have been used to get into the VPN VM but not the others? Only that VM had OpenVPN on it.

  2. What exploit might have been used to take down the ZFS array and passwords in the hypervisor? Is it possible the exploit could have been closed by the upgrades?

  3. It seems to me that if they had root-shell access they could have done a lot more damage than they did. Is the fact that they did not a) because they were being "nice" or b) they didn't really have that level of access?

I realize that these questions can't be answered definitively with the information detail I have here, but any guidance or suggestions from someone who has experience with a similar situation would probably be helpful.

  • 1
    Sorry but almost none of these questions can be answered by anyone here. Could be anything from an OpenVPN 0-day to a leaked password. The only thing I can take a guess at is why they didn't do more damage - they wanted their miner to run and make money. For the rest you'll need to do your own IR process to identify the activities which led to the compromise. Do you have a SIEM? IDS/IPS logs? Forensic image of the disk? – Polynomial Feb 15 '18 at 19:25
2

As you said, it's hard to make good guesses, so this isn't a full answer (it would have been a comment but it's too long for that).

It seems to me that if they had root-shell access they could have done a lot more damage than they did. Is the fact that they did not a) because they were being "nice" or b) they didn't really have that level of access?

In my (limited) experience, attackers of servers often aren't even interested in root access because the things they want to do (set up a spam mailer, a bitcoin miner or use your system as a proxy for various purposes) don't require it. I've been administrating internet-connected servers for 15 years and I haven't yet found an attacker who was interested primarily in the destruction of the system he took over (but as I said, I only have a handful of cases I can look back to).

What exploit might have been used to take down the ZFS array and passwords in the hypervisor? Is it possible the exploit could have been closed by the upgrades?

This is what I would be most worried about. I'd be asking myself how the intruders were able to touch the hypervisor. There's 3 options in my mind:

  1. They attacked the hypervisor through a network service running on the hypervisor (possibly ssh if it's the only one)

  2. They attacked it through the shared ZFS filesystem (depending on what exactly you're sharing in the ZFS, that's very likely or pretty much impossible)

  3. They attacked the VM and then escaped out of the VM into the hypervisor.

Option 3 would give me nightmares. It's possible, there are various demonstrations on security conferences, but I haven't heard of it occuring much in the wild.

What to do

As your questions almost certainly can't be answered by anyone here, you'll have to answer them yourself. You do this by inspecting your systems, looking for the entry point of the malware.

If you can take a snapshot of your VM (or even better, make a copy of it), this is a good way to preserve possible proof of the breakin method. If you can't, you'll have to work on the live VM.

Finding out how an attacker broke into your system is important because if you don't, you can't trust your system to not get infected again. So there's really no way around it.

Once you know how the attackers got in, you'll have to decide whether to install your system from scratch and close the hole, or try to find out what the motives of the attackers were. If you go the second route, you run the risk of guessing wrong and leaving behind a system you can't really trust any more, because it might still contain malicious surprises the attackers left behind.

I'd vote for full reinstall, but sometimes from the things an attacker did on your system, you can tell with high confidence that he was only interested in a single thing (like mining bitcoins), and once you remove that, you can be fairly certain your system is clean again. Still, it's a risk.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.