Malware script uploaded - CPU has been maxed out

Question

I received notifications that the CPU was maxed out. Using Process Manager in WHM, I could see that commands named "fuckyou" was running some kind of cron sript on our server.

The file it was calling was found in a folder I haven't uploaded my self: /home/user/.lesshts/run.sh

This is the content of that file:

#!/bin/sh
#fuckyou ;)
killall -9 kthread > /dev/null 2>&1
kill -9 `pidof kthread`> /dev/null 2>&1
sleep 5
cd /home/user/.lesshts
/home/user/.lesshts/kthread > /dev/null 2>&1

I have now deleted the folder and all files it contained. I've also killed all the processes.

• Have you seen anything similar?
• Apart from overloading the server, can you from the above information understand what the intention of this was? What can it do to our server?
• Most important: Do you have any idea how I can prevent malware from being uploaded to the server again?

Any information is greatly appreciated.

Answer 1

If you use FTP for access to files on your website, you need to be very careful.

If you store your FTP usernames and passwords on your local computer using software like FileZilla, your website can be compromised if malicious software or a Trojan is installed on your computer.

Never store credentials on your local computer.

Additionally, you should use SFTP (Secure FTP) or SSH (Secure Shell) which uses encryption, instead of FTP - FTP transfers credential and information in plain text. This means that any person or program that is “listening” in on the transmission of credentials to the FTP server, can do so relatively easily and then steal these credentials.

Credentials, such as FTP username and passwords, can be compromised by Trojans and viruses installed on the computers of unsuspecting users “sniff” the credentials being transferred over the Internet to the web server.

SSH (Secure Shell) or SFTP (Secure FTP) will avoid credential compromise from “sniffing” attacks.

Answer 2

I have now deleted the folder and all files it contained. I've also killed all the processes.

Did you make copies of the files, or simply delete them? Because without them there's really no way to tell you anything definite about them.

You should not assume you've cleaned everything out or blocked all access to the attacker. Your best bet is to backup your data and wipe the server. If you're not ready to do that, at least walk through steps like those listed here.

Have you seen anything similar?

Google has. Just going by those results, the attack vector was probably Shellshock. Are you patched against it?

Apart from overloading the server, can you from the above information understand what the intention of this was? What can it do to our server?

No, we would need to see the contents of /home/user/.lesshts/kthread to be able to answer that question. If I had to guess, it was probably scanning and trying to infect other people vulnerable to Shellshock, but that's just a wild guess. Again, you removed the obvious sign of compromise. How many non-obvious backdoors did they leave behind?

Updated: Based on the binary you uploaded (see comments below), the kthread program looks like the Tsunami IRC/bot. Here's a good technical analysis. Your system was probably bogged down because it was participating in a DDoS attack.

Most important: Do you have any idea how I can prevent malware from being uploaded to the server again?

Again, there's a good answer on that here. My advice, though, is:

• Build, patch, and secure a replacement system
• Migrate important and necessary data only from old to new system
• Wipe the old system

When you're migrating data, obviously, don't log in from the compromised system to the clean system - copy from the compromised system which has potentially compromised credentials instead.

Answer 3

I see 3 red flags right away :

• first one is that you use WHM, which is a bad idea (just like its little friend cPanel and its colleagues Plesk, Kloxo, etc) - I wouldn't be surprised if there are extreme vulnerabilities in these piles of garbage that would allow root-level access to the server without even compromising any user accounts.

• the second one is that you use FTP... in 2015. In addition to being a horrible and broken protocol right from the beginning, it's also completely insecure and any attacker listening to any client's traffic can get his password. You should use SFTP which is built into OpenSSH, so you have it already.

• finally despite this compromise, you're still trying to fix this nightmare even though the only right thing to do is to nuke it from orbit, reinstall a good distro on it without any kind of web panel.

Now, for a proper answer as to what this code does - after telling you something very nice and friendly, it tries to kill any already running version of itself, it starts itself under the name "ktnread", hoping to fool any inexperienced sysadmins into mistaking it for the real, legitimate kernel thread "kthreadd".

You could actually take a look at what his own kthread is, I'm pretty sure it's some years-old stolen Perl horror story designed to connect to an IRC server and await orders for targets to DoS so the kids who operate these can feel important and powerful, most likely to compensate against what they really are in real life.

To answer your final question, unfortunately it's impossible to completely prevent this from happening but you can at least limit the damage :

• block outgoing connections from the servers/containers/PHP processes and allow them on a case-by-case basis if the application truly needs it - you can use IPtables for that. Even in the case of a compromise, it will prevent their malware from attacking other hosts from your server or sending spam. The script kiddies will even completely give up because their (stolen) malware can't connect to their IRC server and they aren't smart enough to modify it to take orders from the already running HTTP server instead (by monitoring incoming requests on the access log).

• lock the PHP processes into their own chroot or LXC container with only the minimum they need to run - it will be a lot harder for lowlife to successfully deploy their stolen Perl-based malware if there's no /bin/perl. It isn't bulletproof as they could just upload their own Perl binary but it would still help a bit and at least discourage some of them.

• keep the webapps and their plugins up to date, as they're the main entry point for attackers. They often find a file upload vulnerability which allows them to upload all their payload and finally a PHP script that has exec("./payload") in it which they can run just by hitting its URL.

• execute PHP scripts based on a whitelist, as opposed to blindly executing anything ending with .php. Even if there are file upload vulnerabilities, that won't mean instant compromise of the server because they can't get their payload to execute because its path isn't whitelisted (and most file-upload vulnerabilities don't give you the luxury of also specifying the destination path, to override a whitelisted file).

• prevent the PHP process from writing where it's not supposed to - to also block or at least slow down attackers because the file upload vulnerability they found only allows them to upload to an non-standard directory where the PHP user can't write.

• finally, depending on what you are (web host, standard company, etc), you should ask yourself whether you can trust your users. If you're a company the answer is probably yes, just make sure to have good password policies (use keys wherever possible, SSH keys for SFTP for example), and eventually limiting logins to your company's network only. If you're a web host, the answer is definitely no and you should think about enforcing manual verification for new customers (basic background check on the info they provided - whether the street address or the phone is real, or asking for ID) may be a good deterrent for malicious activity.

Bottom line is, you can never be 100% secure, especially in a shared hosting environment. Soon or later, despite all these security recommendations your server will be root-compromised and you'll have to nuke it and reinstall, and depending on your laws/the data hosted on them you may need to inform your customers their data was breached. If the data is anything more valuable than basic blogs I suggest you use separate VMs or bare-metal servers to run the actual code (PHP, etc).

Answer 4

We saw this exact issue on one of our servers today.

Check the cron files on your system to see if they have been manipulated by this malware - in our particular instance the malware was set to run via the apache user's cron. Check if anything has been added to any of your cron files:

grep 'run.sh' /var/spool/cron/*

If any results are found, comment them out or remove them.

With the cron entry as mentioned above, even if we killed all the malware's processes and the malicious kthread daemon, it would restart itself either after a reboot or at 23:59 daily.

Disabling the rogue cron job will at least prevent the daemon from restarting again, but you still need to try and prevent users from being able to infiltrate your system with these files. For example, try to isolate each domain/account so that it runs as its own user account (not apache).

If you are running a shared hosting environment you might consider using something such as CloudLinux or BetterLinux to provide isolation and better security.