I received a spam abuse few days ago on my server. As a precaution, I blocked the SMTP port and started an investigation. I found a running Perl process. The script it was using was deleted but I found its contents in /proc/PID/fd/3. Here's its md5:

server:~# md5sum spam_scipt
c97b6d65fe2e928f190ca4a8cf23747c  spam_scipt

That Perl script was used to send spam from my server. It uses a domain generation algorithm to generate a URL for a dropzone where the status of the spam process is sent. I modified the script in order to see what link is being generated at the moment:

server:~# perl spam_scipt 
wget version is 11
Set browser wget
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
Runned postfix found, use /usr/sbin/sendmail
Probe host redacted.info
Go https://ibkxluxisp.redacted.info:1905//b/index.php?id=5cb885d577c7bbacdae44dd9f7f86b641ad60d58b1b9df07b97953a70376ec47&check=1
Generate host ibkxluxisp.redacted.info:1905

And the second try:

server:~# perl spam_scipt 
wget version is 11
Set browser wget
Warning: bad ps syntax, perhaps a bogus '-'? See http://procps.sf.net/faq.html
Runned postfix found, use /usr/sbin/sendmail
Probe host redacted.info
Go https://wodhvqubux.redacted.info:1905//b/index.php?id=5cb186d575c44b1e5174c7c111636063b338bc3ef4d46e4533036eded038a7&check=1
Generate host wodhvqubux.redacted.info:1905
https://wodhvqubux.redacted.info:1905//b/index.php?id=5a497fe86dde12da162b6460bb8cd215966679ad7bf97338b9b6c2e741fe&version=18&sent=0&user=544c648ace7ff7834db9fadf36bdserver:~# ;

The subdomain only changes now. So after a simple ping we get the IP address of the dropzone:

server:~# ping vodhvqubux.redacted.info
PING vodhvqubux.redacted.info ( 56(84) bytes of data.
64 bytes from ns207415.ovh.net ( icmp_seq=1 ttl=59 time=4.02 ms
--- vodhvqubux.redacted.info ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 4.026/4.026/4.026/0.000 ms

THose criminals have already appeared in public with the same IP address: http://forum.spamcop.net/forums/lofiversion/index.php/t12096.html http://www.jaguarpc.com/forums/general-hosting-network-support/26972-could-need-some-help-compromised-system-cannot-find-door.html

I do not understand how do the upload script using root access without even knowing the password. Here is the environment for that perl process:

SSH_CLIENT=xx.xx.xx.xx.xx 60480 220
SSH_CONNECTION=xxx.xxx.xxx.xxx 60480 xx.xx.xx.xx 220

Maybe that helps. I have upgraded all the software, rebooted server, but that happened again. Also there are no logs for ssh login in access.log.

Perl script http://www.sendspace.com/file/xfrqtt

Can anyone assist with the abuse onto IP from OVH DC? The criminal`s website works just perfect.


2 Answers 2


Your server is infected. It shall be cleansed. With fire.

The primary function of a rootkit is to install itself in an inconspicuous place, and intercept whatever it needs to resist reboots and upgrades. For instance, it may have added its code in the kernel itself, and hijack read and write system calls so that it automatically reinfects the kernel file should it be upgraded.

So the path to your salvation is a complete reinstall of the system from scratch; rebuild partitions, format hard disks, fire up the OS installation CD/DVD. It's the only way to be sure.

(Actually it is conceivable, and has been demonstrated in lab conditions, for some rootkits to install in the motherboard firmware, or the firmware of some peripherals, including keyboards -- as long as such firmware can be reflashed from software, it is theoretically vulnerable. It is rather improbable that your server has been compromised to that depth, so chances are that a complete reinstall will be enough -- no need to escalate to an actual funeral pyre.)

  • I do not think it is a good idea because machine could be infected once more. I think it`s kind off hole in openssh. I have found that script on other servers that have another os, kernel and are not related. Commented Dec 29, 2011 at 14:03
  • @OlierSaari - Your server itself is likely the security hole, its unlikely to be openssh, before some other localized piece of software variable. Start by doing what Tom suggested, if it happens again, eliminate the possible reasons until it stops happening.
    – Ramhound
    Commented Dec 29, 2011 at 15:45
  • ...and this time do disable root ssh access and run fail2ban.
    – symcbean
    Commented Dec 29, 2011 at 16:20
  • How fa2ilban would help? It uses auth.log where there no prints of login attempts. Commented Dec 29, 2011 at 16:24
  • And the main problem is that the same thing appears on different servers that are not related at all. Some of them are even "clean". Commented Dec 29, 2011 at 16:26

As Tom said, you have a rootkit. Your system is compromised, you have to start over again from scratch with known safe files. You can't trust any system utilities, you can't trust your logs -- if all your logs look good now; that doesn't mean they didn't log something interesting when they were attempting to break into your system (but then covered their tracks). I'd boot to a live cd, make a read-only backup of data (not executables/not scripts) or security related files (e.g., /etc/passwd or ssh-keys), and reformat and reinstall your OS.

I sincerely doubt it was a flaw in openssh (if it was this wouldn't be happening on just your servers, but millions of servers worldwide). This is most likely some security flaw unique to your servers.

Maybe you (or one of your users) are reusing passwords and someone intercepted a password that way. Maybe some user has a weak password that can be brute forced (e.g., in common password dictionary lists; or is something basic like a dictionary word and a number), or you gave an account to a malicious user. Maybe someone doesn't have a strong passphrase protecting their ssh private keys and someone got a copy. So change all passphrases and accepted authorized public keys.

Or maybe you installed a malicious application/script or one with major security holes.

Install proactive security measures such as fail2ban (frequent failed login attempts will be slowed down/blocked) / SELinux or AppArmor (access control giving users minimum permissions) / tripwire or OSSEC (integrity check files), etc to reduce the risk of further attacks or at least quickly notify you when you are attacked.

  • I do not understand why it` not a ssh problem. When i move ssh to another port - perl process stops to appear. Also if setup honeypot ssh on 22 port in the logs i see just regular brute from time to time but no successful login attempts. If i move ssh back - i see "perl" running again. If i keep it on 22 port but block connections by ip - no "perl". I do not have explanations. fail2ban is installed. SSH has "AllowUsers root" and only few RSA keys. I have removed them, installed the only mine one-the same story. If i have a rootkit it shouldn`t matter to criminal if i move ssh to another port Commented Jan 1, 2012 at 14:48
  • @OlierSaari - Have you reinstalled your OS? I doubt it was a flaw in OpenSSH that let them initially break into your systems. One of the first things they did once they did get into your system is install a rootkit that replaces safe utilities like OpenSSH's sshd with a compromised version, e.g., that always accepts a malicious key (e.g., compiled into the binary). This compromised version is based on OpenSSH (so it won't seem to be acting differently); so when you switch ports sshd_config the port does change (and apparently their scripts aren't scanning for other ports).
    – dr jimbob
    Commented Jan 2, 2012 at 17:55
  • @OlierSaari - Boot off of a live cd (that you know to be safe) mount the hard disk to /mnt/unsafe_harddrive/ and try running md5sum on the executables of various utilities (e.g., md5sum /mnt/unsafe_harddrive/usr/sbin/sshd) and compare the values to the original values from the operating system (and also check that the version is the same). E.g., ubuntu 11.10 (oneric) has OpenSSH_5.8p1 Debian-7ubuntu1 with a md5sum of 6ea7510e7be723b31af0d586815b6a76 /usr/sbin/sshd. A program like tripwire can do these rootkit checks automatically. Also don't trust md5sum on your corrupted system.
    – dr jimbob
    Commented Jan 2, 2012 at 18:01

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .