1

I received notifications that the CPU was maxed out. Using Process Manager in WHM, I could see that commands named "fuckyou" was running some kind of cron sript on our server.

The file it was calling was found in a folder I haven't uploaded my self: /home/user/.lesshts/run.sh

This is the content of that file:

#!/bin/sh
#fuckyou ;)
killall -9 kthread > /dev/null 2>&1
kill -9 `pidof kthread`> /dev/null 2>&1
sleep 5
cd /home/user/.lesshts
/home/user/.lesshts/kthread > /dev/null 2>&1

I have now deleted the folder and all files it contained. I've also killed all the processes.

  • Have you seen anything similar?
  • Apart from overloading the server, can you from the above information understand what the intention of this was? What can it do to our server?
  • Most important: Do you have any idea how I can prevent malware from being uploaded to the server again?

Any information is greatly appreciated.

  • What possible ways would there be to upload? Do you run FTP or a website or maybe both? There are many ways to get files onto a server ;) – BadSkillz Apr 28 '15 at 9:51
  • I run a website and files can be uploaded using FTP (...if you know the password). – Carl Apr 28 '15 at 9:54
  • Are using FTP or SFTP, SSH ? – Michal Koczwara Apr 28 '15 at 10:24
  • Normally I use FTP. – Carl Apr 28 '15 at 10:27
1

If you use FTP for access to files on your website, you need to be very careful.

If you store your FTP usernames and passwords on your local computer using software like FileZilla, your website can be compromised if malicious software or a Trojan is installed on your computer.

Never store credentials on your local computer.

Additionally, you should use SFTP (Secure FTP) or SSH (Secure Shell) which uses encryption, instead of FTP - FTP transfers credential and information in plain text. This means that any person or program that is “listening” in on the transmission of credentials to the FTP server, can do so relatively easily and then steal these credentials.

Credentials, such as FTP username and passwords, can be compromised by Trojans and viruses installed on the computers of unsuspecting users “sniff” the credentials being transferred over the Internet to the web server.

SSH (Secure Shell) or SFTP (Secure FTP) will avoid credential compromise from “sniffing” attacks.

  • ...if his computer is compromised, it doesn't matter if he stores passwords or not, they could be sniffed at type-time (granted, its easier for some attacks, but not really necessary). It would also be possible for any such Trojan to simply upload a kill-script itself. – Clockwork-Muse Apr 28 '15 at 11:08
  • Yes sure. It could be anything but I always encourage people to use SSH over FTP to avoid similar attacks. – Michal Koczwara Apr 28 '15 at 11:22
  • SSH would only prevent the connection being snooped from outside (and that's the reason you should use it), but still wouldn't protect against on-machine compromise. – Clockwork-Muse Apr 28 '15 at 11:26
  • Well if he will use SSH to connect to his server from his computer instead of FTP and he will not store passwords on his local machine he will stop to spread the infection to his website even if his computer is compromised. – Michal Koczwara Apr 28 '15 at 11:34
  • Absolutely false. Any of the following are possible: Malware sniffs credentials at authentication time (keylogger for passwords) and exfiltrates them, silently adds relevant file/command to stack, malware installs new root certificate/DNS redirect allowing for MITM, prevent the close of the SSH connection and keep using it. The only thing not storing the credentials does is prevent them from being compromised until they're used. Otherwise, anything goes; anything the client machine can do with a person behind it, a rogue process can do too. – Clockwork-Muse Apr 28 '15 at 11:46
1

I have now deleted the folder and all files it contained. I've also killed all the processes.

Did you make copies of the files, or simply delete them? Because without them there's really no way to tell you anything definite about them.

You should not assume you've cleaned everything out or blocked all access to the attacker. Your best bet is to backup your data and wipe the server. If you're not ready to do that, at least walk through steps like those listed here.

Have you seen anything similar?

Google has. Just going by those results, the attack vector was probably Shellshock. Are you patched against it?

Apart from overloading the server, can you from the above information understand what the intention of this was? What can it do to our server?

No, we would need to see the contents of /home/user/.lesshts/kthread to be able to answer that question. If I had to guess, it was probably scanning and trying to infect other people vulnerable to Shellshock, but that's just a wild guess. Again, you removed the obvious sign of compromise. How many non-obvious backdoors did they leave behind?

Updated: Based on the binary you uploaded (see comments below), the kthread program looks like the Tsunami IRC/bot. Here's a good technical analysis. Your system was probably bogged down because it was participating in a DDoS attack.

Most important: Do you have any idea how I can prevent malware from being uploaded to the server again?

Again, there's a good answer on that here. My advice, though, is:

  • Build, patch, and secure a replacement system
  • Migrate important and necessary data only from old to new system
  • Wipe the old system

When you're migrating data, obviously, don't log in from the compromised system to the clean system - copy from the compromised system which has potentially compromised credentials instead.

  • I did save all the files in the .lesshts folder before deleting them. Not sure how to share them with you? My Mac wants to use terminal when opening kthread and I haven't done that. Also, my hosting company says I'm safe when it comes to Shellshock. – Carl Apr 28 '15 at 13:39
  • @Carl open a terminal window and run 'file kthread' on the file. If it says it's an ELF executable, not as trivial. But if it says it's text, bash script, or something like that, put it on pastebin. – gowenfawr Apr 28 '15 at 13:52
  • Sorry, but I'm not very good with terminal and I don't know how to do that.. I added .php to the file name (kthread) and when I opened it, it said ELF (nothing else). Is there anyway I can send you a zipped copy of the folder so that you can have a look at the 8 files? – Carl Apr 28 '15 at 15:29
  • Analyzing a binary (ELF executable) is non-trivial and not something I have time to play with, but you might upload it to VirusTotal on the off chance they'll recognize it. – gowenfawr Apr 28 '15 at 15:41
  • Here is the result of that file - any ideas? virustotal.com/en/file/… – Carl Apr 28 '15 at 16:20
0

I see 3 red flags right away :

  • first one is that you use WHM, which is a bad idea (just like its little friend cPanel and its colleagues Plesk, Kloxo, etc) - I wouldn't be surprised if there are extreme vulnerabilities in these piles of garbage that would allow root-level access to the server without even compromising any user accounts.

  • the second one is that you use FTP... in 2015. In addition to being a horrible and broken protocol right from the beginning, it's also completely insecure and any attacker listening to any client's traffic can get his password. You should use SFTP which is built into OpenSSH, so you have it already.

  • finally despite this compromise, you're still trying to fix this nightmare even though the only right thing to do is to nuke it from orbit, reinstall a good distro on it without any kind of web panel.

Now, for a proper answer as to what this code does - after telling you something very nice and friendly, it tries to kill any already running version of itself, it starts itself under the name "ktnread", hoping to fool any inexperienced sysadmins into mistaking it for the real, legitimate kernel thread "kthreadd".

You could actually take a look at what his own kthread is, I'm pretty sure it's some years-old stolen Perl horror story designed to connect to an IRC server and await orders for targets to DoS so the kids who operate these can feel important and powerful, most likely to compensate against what they really are in real life.

To answer your final question, unfortunately it's impossible to completely prevent this from happening but you can at least limit the damage :

  • block outgoing connections from the servers/containers/PHP processes and allow them on a case-by-case basis if the application truly needs it - you can use IPtables for that. Even in the case of a compromise, it will prevent their malware from attacking other hosts from your server or sending spam. The script kiddies will even completely give up because their (stolen) malware can't connect to their IRC server and they aren't smart enough to modify it to take orders from the already running HTTP server instead (by monitoring incoming requests on the access log).

  • lock the PHP processes into their own chroot or LXC container with only the minimum they need to run - it will be a lot harder for lowlife to successfully deploy their stolen Perl-based malware if there's no /bin/perl. It isn't bulletproof as they could just upload their own Perl binary but it would still help a bit and at least discourage some of them.

  • keep the webapps and their plugins up to date, as they're the main entry point for attackers. They often find a file upload vulnerability which allows them to upload all their payload and finally a PHP script that has exec("./payload") in it which they can run just by hitting its URL.

  • execute PHP scripts based on a whitelist, as opposed to blindly executing anything ending with .php. Even if there are file upload vulnerabilities, that won't mean instant compromise of the server because they can't get their payload to execute because its path isn't whitelisted (and most file-upload vulnerabilities don't give you the luxury of also specifying the destination path, to override a whitelisted file).

  • prevent the PHP process from writing where it's not supposed to - to also block or at least slow down attackers because the file upload vulnerability they found only allows them to upload to an non-standard directory where the PHP user can't write.

  • finally, depending on what you are (web host, standard company, etc), you should ask yourself whether you can trust your users. If you're a company the answer is probably yes, just make sure to have good password policies (use keys wherever possible, SSH keys for SFTP for example), and eventually limiting logins to your company's network only. If you're a web host, the answer is definitely no and you should think about enforcing manual verification for new customers (basic background check on the info they provided - whether the street address or the phone is real, or asking for ID) may be a good deterrent for malicious activity.

Bottom line is, you can never be 100% secure, especially in a shared hosting environment. Soon or later, despite all these security recommendations your server will be root-compromised and you'll have to nuke it and reinstall, and depending on your laws/the data hosted on them you may need to inform your customers their data was breached. If the data is anything more valuable than basic blogs I suggest you use separate VMs or bare-metal servers to run the actual code (PHP, etc).

  • Slight correction: there's no kernel process named "kthread". The kernel process is named "kthreadd". I'd guess the "killall" is an upgrade process for the malware. – Mark Apr 29 '15 at 7:42
  • @Mark thanks, corrected. Pretty stupid for them to update their malware through the same hole they used to get in rather than silently updating itself via the already running malware. – user42178 Apr 29 '15 at 7:53
0

We saw this exact issue on one of our servers today.

Check the cron files on your system to see if they have been manipulated by this malware - in our particular instance the malware was set to run via the apache user's cron. Check if anything has been added to any of your cron files:

grep 'run.sh' /var/spool/cron/*

If any results are found, comment them out or remove them.

With the cron entry as mentioned above, even if we killed all the malware's processes and the malicious kthread daemon, it would restart itself either after a reboot or at 23:59 daily.

Disabling the rogue cron job will at least prevent the daemon from restarting again, but you still need to try and prevent users from being able to infiltrate your system with these files. For example, try to isolate each domain/account so that it runs as its own user account (not apache).

If you are running a shared hosting environment you might consider using something such as CloudLinux or BetterLinux to provide isolation and better security.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.