23

So I've been fighting this problem for months now and decided that it's beyond my limited (if at all) server skills, and that I need help from the pros.

I have a VPS (with root access) which hosts several different PHP websites, some of which are WordPress-based. Some of the sites got infected with the a malware as a result of the MailPoet vulnerability. I cleaned the infected sites, completely removed MailPoet, backdoor accounts, and related stuff, but the malware keeps resurrecting once in a while. Below is what I can describe about it:

  • There are two malware signatures (sorry if I'm using the wrong term), both are injected at the very top of PHP pages. Once looks like this <?php $ozufdqjmhx = '7825h!>!%x5c%x7825tdz)%x5c%x7825bbT-%x5c%x782vg}... with the variable $ozufdqjmhx changes from time to time, the other begins with <?php if(!isset($GLOBALS[\'\a\e\0... etc etc
  • The malware comes back at random intervals. Sometimes it comes back a day after cleaning, sometimes a week, or several weeks.
  • Only previously infected files/directories/websites get infected again. New directories, or old unaffected ones, are always clean. New files in old infected directories though, get infected.
  • maldet (using ClamAV I believe) can't detect any malware. PHP Shell Detector can, but it cannot fix due to being a detector only.

Can you guys help, or give a direction I should be heading to? A million thanks in advance!

(Also I'm sorry if this question doesn't fit the site's regulations. When I'm a daily user of StackOverflow, this is my first time on this Security subsite).

EDIT: I really appreciate any recommendation from you guys, but wiping the sever and start from scratch is not an option. If it was, why would I ask this question to begin with, right? :)

EDIT 2: Following @Mints97's answer, I've checked all the open ports -- looks normal:

21/tcp    open  ftp
22/tcp    open  ssh
25/tcp    open  smtp
53/tcp    open  domain
80/tcp    open  http
110/tcp   open  pop3
143/tcp   open  imap
443/tcp   open  https
465/tcp   open  smtps
587/tcp   open  submission
993/tcp   open  imaps
995/tcp   open  pop3s
3000/tcp  open  ppp
3306/tcp  open  mysql
5432/tcp  open  postgresql
8000/tcp  open  http-alt
8080/tcp  open  http-proxy
8082/tcp  open  blackice-alerts
10000/tcp open  snet-sensor-mgmt
20000/tcp open  dnp

EDIT 3: This is for @QuestionOverflow: When searching for the 4 domains you mentioned in your other answer, I came across a script to eliminate the malware here. In the code we can see if (preg_match('/^<\?php \$[a-z]{10} = \'/', $fh_str)) {, which targets EXACTLY the first signature. I would say now it's the same malware, or at least from the same guy via the same vulnerability. Pretty interesting.

EDIT 4: The second malware has already been discussed here if it may help, and yes, apparently both fetch some payload randomly from 4 domains: "33db9538.com", "9507c4e8.com", "e5b57288.com", "54dfa1cb.com". I've added all 4 into my hosts files, pointing to 127.0.0.1. Let's see what's next.

EDIT 5: Several suggest that this question has already been answered here at How do you explain the necessity of “nuke it from orbit” to management and users?. Honestly I fail to see how the other question answers mine. I'm asking how to eliminate a malware, not to explain to my boss why I should reinstall a server.

marked as duplicate by D.W., schroeder, Jens Erat, Stephane, RoraΖ Feb 5 '15 at 12:47

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 3
    In regards to not wiping the server - you may not have a choice if the server has had a rootkit installed. With a rootkit the server is essentially no longer under your control anymore. Depending on the rootkit there may be no way to take it back other than wiping the server and reinstalling. – Vegard Feb 4 '15 at 10:01
  • Let's hope it's not a rootkit then :) I've scanned the system using several rootkit tools and the results are all clean. – An Phan Feb 4 '15 at 11:29
  • 5
    @AnPhan in case of a targeted attack the malware will be a custom one and won't be detected by any scanning tools because they rely on signatures of already known malware. Plus, a rootkit can hide itself from any scan tool running on the same system. – user42178 Feb 4 '15 at 12:22
  • 1
    That second script looks similar to this one and does not contain a backdoor. I am curious about the first one though. – Question Overflow Feb 4 '15 at 14:35
  • 2
    The canonical answer to this kind of question is: "nuke it from orbit, it's the only way to be sure". In other words, wipe the entire machine, re-install from known-good backups, change all your passwords, update all services to their latest versions with all available security patches, etc. -- and then apply good security practices and hardening to your machine to avoid another compromise in the future. It's a pain, but there's a good reason why security folks recommend it -- as you're discovering, the alternative is often even more painful. Are you sure you can't do that? – D.W. Feb 4 '15 at 23:29
23

I would enable auditd to monitor changes to the files you expect to be backdoored. You will be able to determine which account and process that is responsible for doing these changes.

After installing auditd (not installed pr default on all systems), you can start monitoring changes in files. To do this, simply run the command:

auditctl -w /var/www -p wa

This command will log all file changes to the auditd log file. Usually you will find it in /var/log/audit/, but it is system dependent.

If you are concerned that the attacker might notice this (and maybe remove the rule), you can lock it by running the command:

auditctl -e 2

No further changes can be made to the audit rules until the system is rebooted. Before doing this, be sure that the audit rule above do not generate an insane amount of logs.

Lastly, the audit logs are a bit cryptic. Once you have found that the system is backdoored again, you can search for the audit trail, using the command:

ausearch -f /var/www/backdoored_file.php

I hope this will provide you with the clues on what is going on. Good luck!

Additional:

To make the audit rules survive reboot, you have to define them in /etc/audit/audit.rules

  • 1
    Nice details. I'll try and update during the process. Thanks! – An Phan Feb 4 '15 at 9:57
  • @AnPhan - I wonder out of curiosity, did if you finally figured out what it was backdooring your php scripts? – Dog eat cat world Jun 9 '16 at 21:06
9

I originally posted this as a comment, but I think this could do with a little explanation.

From my experience with website takeover scenarios, when a shell is uploaded to a website, the hacker either manages to exploit a vulnerability in the server, gain root access, backdoor your SSH and compromise all other sites on the server, or he simply doesn't manage this and makes do with a simple shell. I think that, in your case, it is the second scenario, given that you say that only one website on the VPS was infected.

Now for how does the shell "ressurect". If your server wasn't "rooted", that leaves only four other options I can think of:

  1. a "bindport" backdoor
  2. a "backconnect" backdoor (rare)
  3. a custom backdoor or a "downloader" in one of your PHP files
  4. a compromised MySQL account with file access and remote login priveliges.

Let's talk about the first two and how to deal with them first. "bindport" and "backconnect" are two small programs, usually Perl scripts, that are traditionally shipped with web shells. They are usually created in (and executed from) the /tmp folder, which is writable to everything. To find them, you can monitor all of your processes for weird scripts or programs and have a good look in the /tmp folder. Also, it is advisable to set up a firewall.

"Bindport" opens a new port for incoming connections and provides Unix shell access to anyone who knocks in (it's usually password-protected). To find it, look for weird open ports (many hackers just have it open port 31337 or something like that).

"Backconnect" does exactly what it is called - it opens a connection to a remote server, also granting shell access. It is used more rarely than "bindport", mainly because most hackers are too lazy to bother with using it. They normally resort to this method only if "bindport" fails for some reason (like firewall settings).

Now, about the custom backdoors and "downloaders". These are seldom used, because the attacker needs to know PHP at least a bit to use it (and most site-jackers today are nothing more than skriptkiddies or badly-made bots). They're mostly stand-alone files like simplistic shells, or a couple of extra lines of code injected into one of your scripts. They either execute commands given to them (PHP code, shell commands) or do a simple file write with data pased to them (which can be another backdoor). You can try looking for files that have PHP code like eval, preg_replace with the e modifier, exec, system, fopen/fwrite and other file functions, etc. However, the best and most certain way to deal with this stuff is simply restoring the entire website from a backup. If you decide to do this, make sure you wipe all other of the site's files from the server beforehand, just in case you missed a stand-alone shell or backdoor.

And the last highly improbable, but still possible case. If you've been running WordPress with the MySQL root user (or just a user with file write rights), or simply with a user that had enough rights to view the password hash of another user with file write rights, and that user with file write rights has the privelige to connect to the database from anywhere, well... you get it. I'd recommend changing all your MySQL passwords.

Now to why only a certain set of directories gets infected. There are two possible cases: either the hacker, having failed to "root" the server, make do with the limited amount of directories available to them (probably all of your website's directories), or they have access to other directories but simply don't use them.

If it is the latter case, I'd wager that either you're dealing with an idiot, or someone extremely lazy. Or with a bot. Yes, this may disappoint you, but I doubt that your site is really important enough for an experienced hacker: it might be used to grab a few clicks from your users, to host malware or simply churn out a metric ton of spam a couple of times. You're either dealing with a scriptkiddie, or an inexperienced fool trying to scrape a couple of cents from click-trading and spam, or with a bot. However, I think it is more likely a living person: that'd explain the irregularity of the shell's "ressurections".

  • I can't say how much I appreciate your answer as well as your time spent writing it. Have edited the question with the port scan result. – An Phan Feb 4 '15 at 15:35
  • @AnPhan: no problem. You can now try checking what's behind these open ports with sudo lsof -i tcp:port-id. I'd check 8082 first thing, but you're probably not bind-ported. You can check my other suggestions, too. Also, note that I'm not suggesting to "wipe the server", just the compromised website would do nicely. Of course, if you have backups (I really hope you do). – Mints97 Feb 4 '15 at 15:44
  • Thanks again, will check right now. I do have backups and normally a git reset --hard will reset my websites nicely. What really annoying is the "resurrection" part. – An Phan Feb 4 '15 at 15:49
  • Also, 8082 is my uptime script. 10000 is Webmin, and 20000 is usermin. – An Phan Feb 4 '15 at 15:57
  • Since you're using git, check your commit history to see if the backdoor hasn't wiggled it's way into your commit history. – Ohnana Feb 4 '15 at 15:58
6

I've had a very similar thing happen to a site I manage. After much frustration of deleting the malicious code and then it appearing about 2 weeks later, I discovered this:

I took note of the date stamp of when all the files got modified, then I looked up the access log for that minute. I saw a certain page was requested that seemed suspicious, since it was a 404.php of a wordpress theme that wasn't active. I checked that page and saw one line which was basically eval($_POST['php']) encoded in base64.

So I didn't delete the code, instead I changed it save a log of anything that gets sent to post of that page. Sure enough 2 weeks later, my site remained safe, but a log file recorded some interesting code sent to it.

  • Nice. I've set up auditd to keep track of changes, following the top answer. Once changes are detected, hopefully I'll have a chance to try your method. – An Phan Feb 5 '15 at 2:53
  • 1
    I mentioned that in my answer, too... – Mints97 Feb 5 '15 at 7:58
4

It sounds like the attackers have installed a rootkit on your server. A rootkit can provide a backdoor even if everything looks clean.

The best approach now in my opinion would be to wipe the server and reinstall from scratch. Patch the websites to eliminate the vulnerability. If you need to restore from backup (you have backups, right? :) ) make sure it's a clean one.

Set up a script to look for the traces of infection (for instance the ID 10001 user) in case there are some other vulnerabilities around in addition to the MailPoet problem.

  • ^This is what I'd do. After restoration... Ensure that you haven't left any input areas vulnerable. Check your textfields by writing (remove brackets): ('<script>alert(1);</script>) to see if they can inject javascript (which can then paste in some php etc... also ensure every ('".|&-${}<>/) are escaped. Ensure you are running the latest security updates on the system. Make sure server configs are default. do research on vulnerabilities before modifying server configs; Allow override none, image comment removal, cookies, injection. NEVER click on links in emails unless its from a known source. – josh.thomson Feb 4 '15 at 10:31
1

If you don't know how you got infected in the first place, then my guess would be the resurrection is coming from you getting hit by the same vulnerability scanner that hit you the first time round - if this is the case, following @scrollup's advice to compare file timestamps to web traffic should find you the vulnerable script, as well as giving you some good IPs to block from your site.

In my experience, at least when targeting Apache, page-modifying scripts will target a very specific subset of the folders on your machine: those that they can find in your /etc/httpd/conf/httpd.conf folder. I had some other web folders defined in included files, but the script that searched for the folders to inject was clearly only interested in grepping the folders from the one file.

So, see if there's anything significantly different in your Apache config between those sites that were impacted, and those that were not. It may give you a good approach to protect most of your sites from it in future, while leaving one "canary" or "honeypot" site that you can monitor for recurrences.

  • 1
    Thanks. Since we're using Webmin, /etc/httpd/conf/httpd.conf contains all of our site records (that's how Virtualmin works). Anyway yesterday I've disabled/deleted quite a few obsolete files, similar to your recommendation of narrowing the targets. Let's see how it goes. – An Phan Feb 5 '15 at 2:33
1

This might seem like a simple question, but have you checked for anything suspicious in you /tmp directory? While different variants of malware are all over the place, many commonly take advantage of system access bugs—such as the bash shellshock bug—to plant executable code right in the /tmp directory.

If you are unsure what should/shouldn’t be in /tmp/ there is an easy—but extreme—thing you can do to clear out the bad stuff. Just run this online in the command line:

rm -rf /tmp && mkdir /tmp && chown root:root /tmp && chmod 1777 /tmp

Or run each command individually like this:

sudo rm -rf /tmp 
sudo mkdir /tmp
sudo chown root:root /tmp
sudo chmod 1777 /tmp

Then reboot the server to see if that clears things up. If it does, congrats! But you are not out of the woods yet since it whatever caused the original system infection might still penetrate your system, it’s only a matter of time before they reinfect you again. Meaning, hopefully this will clean up the mess caused by a weakness in your system you hopefully have plugged up already. But you need to to be sure to find out what that weak-point might have been and be sure to harden it.

I would also recommend checking the crontab entries for the root user via a simple crontab -l like this:

sudo crontab -l

And perhaps even running the bash script taken from this answer on Stack Overflow which will give you a nice, overview report of all crontabs installed on the system:

#!/bin/bash

# System-wide crontab file and cron job directory. Change these for your system.
CRONTAB='/etc/crontab'
CRONDIR='/etc/cron.d'

# Single tab character. Annoyingly necessary.
tab=$(echo -en "\t")

# Given a stream of crontab lines, exclude non-cron job lines, replace
# whitespace characters with a single space, and remove any spaces from the
# beginning of each line.
function clean_cron_lines() {
    while read line ; do
        echo "${line}" |
            egrep --invert-match '^($|\s*#|\s*[[:alnum:]_]+=)' |
            sed --regexp-extended "s/\s+/ /g" |
            sed --regexp-extended "s/^ //"
    done;
}

# Given a stream of cleaned crontab lines, echo any that don't include the
# run-parts command, and for those that do, show each job file in the run-parts
# directory as if it were scheduled explicitly.
function lookup_run_parts() {
    while read line ; do
        match=$(echo "${line}" | egrep -o 'run-parts (-{1,2}\S+ )*\S+')

        if [[ -z "${match}" ]] ; then
            echo "${line}"
        else
            cron_fields=$(echo "${line}" | cut -f1-6 -d' ')
            cron_job_dir=$(echo  "${match}" | awk '{print $NF}')

            if [[ -d "${cron_job_dir}" ]] ; then
                for cron_job_file in "${cron_job_dir}"/* ; do  # */ <not a comment>
                    [[ -f "${cron_job_file}" ]] && echo "${cron_fields} ${cron_job_file}"
                done
            fi
        fi
    done;
}

# Temporary file for crontab lines.
temp=$(mktemp) || exit 1

# Add all of the jobs from the system-wide crontab file.
cat "${CRONTAB}" | clean_cron_lines | lookup_run_parts >"${temp}" 

# Add all of the jobs from the system-wide cron directory.
cat "${CRONDIR}"/* | clean_cron_lines >>"${temp}"  # */ <not a comment>

# Add each user's crontab (if it exists). Insert the user's name between the
# five time fields and the command.
while read user ; do
    crontab -l -u "${user}" 2>/dev/null |
        clean_cron_lines |
        sed --regexp-extended "s/^((\S+ +){5})(.+)$/\1${user} \3/" >>"${temp}"
done < <(cut --fields=1 --delimiter=: /etc/passwd)

# Output the collected crontab lines. Replace the single spaces between the
# fields with tab characters, sort the lines by hour and minute, insert the
# header line, and format the results as a table.
cat "${temp}" |
    sed --regexp-extended "s/^(\S+) +(\S+) +(\S+) +(\S+) +(\S+) +(\S+) +(.*)$/\1\t\2\t\3\t\4\t\5\t\6\t\7/" |
    sort --numeric-sort --field-separator="${tab}" --key=2,1 |
    sed "1i\mi\th\td\tm\tw\tuser\tcommand" |
    column -s"${tab}" -t

rm --force "${temp}"

As complex as malware infections might seem, once you know where the common choke-points are, you can focus your efforts to purge your system of that junk once and for all.

  • Thanks for the answer, and the great script. I did try to clean /tmp and checked through the crontab -- again, nothing suspicious. – An Phan Feb 5 '15 at 2:30
  • @AnPhan Well, are you sure your base OS is patched? Specifically bash itself in light of the bash “shellshock” bug? Check out my answer on Super User over here to see what I mean. While you say the malware is PHP-based, in many cases the initial intrusion vector is something else like an unpatched version of bash. – JakeGould Feb 5 '15 at 2:39
  • Yes it's patched. Or at least the shellshock script test returns hello. – An Phan Feb 5 '15 at 2:46
  • @AnPhan Well, if you are unsure about bash please read the contents of this site and test other bash related flaws that have been found. – JakeGould Feb 5 '15 at 2:55
  • Thanks. Ran the test from the site, the result is all clean. – An Phan Feb 5 '15 at 2:59
0

There isn't enough information in your post to determine thie cause of this. You'll have to do some investigating. The first thing to check is the creation/modification time of the files and then compare that to your access and ftp logs around the same time in order to determine how the files were modified. This will hopefully lead you to some /plugin/.shell.php file that is the left over backdoor from the initial infection. If your logs don't appear to contain anything useful it is possible that a backdoor account has been created in your WP install and what you are seeing is the result of "normal operations" by a malicious user.

  • Thanks for your answering. I forgot to mention that I cleaned all backdoor accounts too (the MailPoet malware indeed created a user with ID 10001 in each infected site's database). – An Phan Feb 4 '15 at 7:05
0

Why not set a watch on the creation of the new user? And once that happens, you send an offsite mail to inform you of the creation.

Not the answer you're looking for? Browse other questions tagged or ask your own question.