Perl script rootkit (exploit)

Question

I received a spam abuse few days ago on my server. As a precaution, I blocked the SMTP port and started an investigation. I found a running Perl process. The script it was using was deleted but I found its contents in /proc/PID/fd/3. Here's its md5:

server:~# md5sum spam_scipt
c97b6d65fe2e928f190ca4a8cf23747c  spam_scipt

That Perl script was used to send spam from my server. It uses a domain generation algorithm to generate a URL for a dropzone where the status of the spam process is sent. I modified the script in order to see what link is being generated at the moment:

server:~# perl spam_scipt 
wget version is 11
Set browser wget
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
Runned postfix found, use /usr/sbin/sendmail
Probe host redacted.info
Go https://ibkxluxisp.redacted.info:1905//b/index.php?id=5cb885d577c7bbacdae44dd9f7f86b641ad60d58b1b9df07b97953a70376ec47&check=1
Generate host ibkxluxisp.redacted.info:1905
https://ibkxluxisp.redacted.info:1905//b/index.php?id=5fb58ad575c14b08785ae5255ffbf83c9f561d18e961b2eb96dc5a058a41&version=18&sent=0&user=584e6388c671f38756eac21cec

And the second try:

server:~# perl spam_scipt 
wget version is 11
Set browser wget
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
Runned postfix found, use /usr/sbin/sendmail
Probe host redacted.info
Go https://wodhvqubux.redacted.info:1905//b/index.php?id=5cb186d575c44b1e5174c7c111636063b338bc3ef4d46e4533036eded038a7&check=1
Generate host wodhvqubux.redacted.info:1905
https://wodhvqubux.redacted.info:1905//b/index.php?id=5a497fe86dde12da162b6460bb8cd215966679ad7bf97338b9b6c2e741fe&version=18&sent=0&user=544c648ace7ff7834db9fadf36bdserver:~# ;

The subdomain only changes now. So after a simple ping we get the IP address of the dropzone:

server:~# ping vodhvqubux.redacted.info
PING vodhvqubux.redacted.info (94.23.208.20) 56(84) bytes of data.
64 bytes from ns207415.ovh.net (94.23.208.20): icmp_seq=1 ttl=59 time=4.02 ms
^C
--- vodhvqubux.redacted.info ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.026/4.026/4.026/0.000 ms

THose criminals have already appeared in public with the same IP address: http://forum.spamcop.net/forums/lofiversion/index.php/t12096.html http://www.jaguarpc.com/forums/general-hosting-network-support/26972-could-need-some-help-compromised-system-cannot-find-door.html

I do not understand how do the upload script using root access without even knowing the password. Here is the environment for that perl process:

SHELL=/bin/bash
SSH_CLIENT=xx.xx.xx.xx.xx 60480 220
USER=root
MAIL=/var/mail/root
PATH=/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin
PWD=/tmp
SHLVL=1
HOME=/root
LOGNAME=root
SSH_CONNECTION=xxx.xxx.xxx.xxx 60480 xx.xx.xx.xx 220
_=/usr/bin/nohup
OLDPWD=/root

Maybe that helps. I have upgraded all the software, rebooted server, but that happened again. Also there are no logs for ssh login in access.log.

Perl script http://www.sendspace.com/file/xfrqtt

Can anyone assist with the abuse onto IP from OVH DC? The criminal`s website works just perfect.

Answer 1

Your server is infected. It shall be cleansed. With fire.

The primary function of a rootkit is to install itself in an inconspicuous place, and intercept whatever it needs to resist reboots and upgrades. For instance, it may have added its code in the kernel itself, and hijack read and write system calls so that it automatically reinfects the kernel file should it be upgraded.

So the path to your salvation is a complete reinstall of the system from scratch; rebuild partitions, format hard disks, fire up the OS installation CD/DVD. It's the only way to be sure.

_{(Actually it is conceivable, and has been demonstrated in lab conditions, for some rootkits to install in the motherboard firmware, or the firmware of some peripherals, including keyboards -- as long as such firmware can be reflashed from software, it is theoretically vulnerable. It is rather improbable that your server has been compromised to that depth, so chances are that a complete reinstall will be enough -- no need to escalate to an actual funeral pyre.)}

Answer 2

As Tom said, you have a rootkit. Your system is compromised, you have to start over again from scratch with known safe files. You can't trust any system utilities, you can't trust your logs -- if all your logs look good now; that doesn't mean they didn't log something interesting when they were attempting to break into your system (but then covered their tracks). I'd boot to a live cd, make a read-only backup of data (not executables/not scripts) or security related files (e.g., /etc/passwd or ssh-keys), and reformat and reinstall your OS.

I sincerely doubt it was a flaw in openssh (if it was this wouldn't be happening on just your servers, but millions of servers worldwide). This is most likely some security flaw unique to your servers.

Maybe you (or one of your users) are reusing passwords and someone intercepted a password that way. Maybe some user has a weak password that can be brute forced (e.g., in common password dictionary lists; or is something basic like a dictionary word and a number), or you gave an account to a malicious user. Maybe someone doesn't have a strong passphrase protecting their ssh private keys and someone got a copy. So change all passphrases and accepted authorized public keys.

Or maybe you installed a malicious application/script or one with major security holes.

Install proactive security measures such as fail2ban (frequent failed login attempts will be slowed down/blocked) / SELinux or AppArmor (access control giving users minimum permissions) / tripwire or OSSEC (integrity check files), etc to reduce the risk of further attacks or at least quickly notify you when you are attacked.