Server compromised for 2nd time, cannot locate source of attack

Question

I need some help tracing a vulnerability on my server. For the second time, my server has been compromised with files being replaced with virus-ridden downloads.

According to the filesystem dates, over a period of 45 minutes 4 exe files on my server were replaced with renamed versions of the same virus.

My web server is running Ubuntu 10.4.3 LTS with kernel version 2.6.32-31-generic, kept completely patched and up to date.

The only way of accessing the shell is via SSH and a password-protected private key that I have with me on a USB stick. Password SSH login is disabled and the server logs (which I know can be modified, but I have good reason to believe they haven't) indicate that SSH was not used to log into the server.

The web serving software stack is very complicated. There's PHP (5.3.3-1) w/ Suhosin v0.9.29, nginx 1.0.9 (now updated to 1.0.10), Tomcat (in a jail and I suspect not associated), and MySQL 5.1.41.

I admit that at the time of the first attack, I had been content to blithely chmod -R 777 my web directory for headache-mitigation purposes. Now I run a complete mess of PHP scripts including but not limited to WordPress, vBulletin, and several homemade products; the first two of which are always up to date and the latter has been written with fairly great care to escape or normalize any user-inputted values.

Given the weak file permissions but strongly-locked down server access, I was highly tempted to suspect a vulnerability in one of the many PHP scripts that allowed the execution of random code.

I have since completely locked down the file permissions. nginx/php both run as www-data:www-data with all files given only execute and read permissions (chmod -R 550 /var/www).

Yet today, after all this, my server was again compromised.

The thing is, the files that were replaced still have 550 permissions, the SSH logs indicate no log in, and I'm completely lost as to what to do or try next.

I tried to recreate the attack on the paths that were replaced with a very basic PHP script:

$file = fopen('/var/www/mysite.com/path/to/file', 'w');
fwrite($file, 'test');
fclose($file)

But that gave me the appropriate permissions denied error.

Can anyone please, please advise me where to look next for the source of this vulnerability? Am I missing something with my file permissions?

I know that server, once compromised, is pretty much "gone" forever. But that's not really an option here. I've recursively grepped my entire /var/log folder for the afflicted file names hoping to find something, but nothing came up.

I also searched for any scripts in the cron folder or elsewhere that might have been placed at the time of the first attack to attack again at a later date, but (a) found nothing, and (b) shouldn't find anything as the files in /etc/ are not modifiable by www-data (assuming a nginx/PHP point of infiltration).

I should add that both times I have grep'd the nginx access logs (combined style) for the names of the infected files, but found nothing. I do understand/realize that many ways of obscuring the file names from my greps exist, however.

Answer 1

Some techniques for trying to find how your attacker got in:

1. Look at the timestamps on any files you know the attacker changed then look through all your logs for entries as close to each timestamp as possible. As others have said, the web access logs and web error logs are the most likely to hold the evidence of the original attack vector but other log files may hold clues as well. Error logs often hold the best clues. The logs of all network-accessible daemons are also good places to look.
2. It may also be worthwhile looking for other files with timestamps close to the ones you know. /etc/passwd is an obvious one but even log files can be suspicious if they have an unusual timestamp. If logrotate runs at the same time every day and one of your log files has a timestamp that doesn't match this time, it was probably altered to cover his tracks, and now you know a little more about what he did. Don't forget the .bash_history files in user's home directories. the find command should be able to handle this for you.
3. Run scalp over your web access logs. The original attack may have happened some time before the files appeared. Scalp will produce false positives but it should narrow down the potential suspect log entries for you to manually analyse for irregularities. It may also miss the attack entirely - it's not perfect - but it may help.

Don't spend too much time on forensics in the compromised system. As others have noted, the attacker has had the opportunity to remove all evidence of the original attack and to add a rootkit to hide his continued presence. If he has missed something or if he has not even tried, the above tasks may work but if he has hidden himself well then you would be just wasting valuable time.

If you fail to find the source of the attack, wipe and re-install the server in question but with some additions in order to catch him next time.

1. Ship your logs off to a different server using some variant of syslog. (rsyslog and syslog-ng are the recommended choices) Preferably, this server should do nothing but receive logs and should not share login keys or passwords with any other server. The goal is to make sure your logs cannot be tampered with or deleted.
2. Add extra logging beyond the default. Jeff already mentioned AppArmor and since you are using Ubuntu this will probably be the best choice. Make sure its logs are sent to the logging box.
3. Install and turn on the audit daemon. Make sure its logs are sent to the logging box.
4. Run an IDS such as Snort, PHP-IDS or mod_security. (or more than one, these tool don't all do exactly the same job) Some hardware firewalls come with IDS/IPS modules. Make sure the logs from the IDS are sent to the logging box.
5. Add a file integrity monitoring system such as AIDE or Tripwire. Make sure the logs from these tools are sent to the logging box.
6. Monitor your logs. Falling short of a commercial SIEM system, something like Splunk can be installed for free and can analyse a limited quantity of logs. Set up rules to match what is normal for your servers and filter them out. Whatever is left is worthy of closer inspection.

There is much more you can do if you have the time and the money, such as full network packet captures, but realistically this is about all a lone sysadmin can be expected to handle. If the attacker shows up again, you will be much more likely to find the attack vector and much more likely to detect him as soon as the attack is made.

Answer 2

@Iszi is absolutely correct here. You need to completely wipe and rebuild, as a good rootkit will prevent you seeing any evidence of its existence.

Otherwise there is a strong likelihood that anything you do now is pointless. In any case, you can no longer trust the server.

Answer 3

Muhammad, in some cases I saw the attack was conducted by a bot that exploited a vulnerability on the system (Usually PhpMyAdmin) and uploaded files to the server (mainly /tmp). What happens is that those files stay there and admin won't notice until the attacker go over their bot logs and start building on the first compromised.

It is quite possible that a vulnerability in PhpMyAdmin or vBulletin or some script you are using was exploited. Later on you updated the vulnerable app, but the compromise already happened.

If you don't do a rebuild, what happens is that files left by the attacker were used to compromise your system, again. Installing OSSEC on your system and monitoring critical files/folders will help you detecting such activity and allows you to take action more quickly.

Answer 4

Just to add a couple of thoughts to the comments already provided. As @symcbean says, third party code is a likely source of issues.

If I was to hazard a guess, I'd look at the Wordpress application and associated plugins as being a likely source of your compromise. There have been a large number of security issues discovered in Wordpress plugins recently and many of them are being actively exploited, so if you have that installed and not 100% up to date you could well have issues.

In terms of addressing the Wordpress piece of your problem, you could look at wpscan which is a wordpress security scanner and also the secure wordpress plugin

Answer 5

Usually when people say that they can't afford to wipe a server, they mean that the time to reset the configuration will be too much. However, there's a way to go about this and not have to find the vulnerability immediately -- though you must replace compromised files. I suggest implementing security before exposing your clean files to the Internet. What broke them once will break them again.

Steps below are Debian-focused (you didn't say which you had, so I took my own preference), but you can perform them on Red Hat systems as well. On a Red Hat based system, you may have an easier time substituting AppArmor suggestions with corresponding SELinux setup. Change "dpkg" to matching "yum" commands.

• Build a new system using the packages you already have installed. dpkg --get-selections will give you a list that can be piped into the new system using dpkg --set-selections.

• Configure apparmor to heavily restrict nginx and any other daemons (Tomcat, etc.) used by it. Allow reading only of configuration and web page files. Some writing might need to be enabled to the web directory depending on what you're doing. Make sure that your configuration forces any child processes to inherit the Apache restrictions.

• Copy everything else back.

Pay very close attention to your AppArmor logs. When you see Apache try to violate its permissions, you can correlate that with your HTTP access / error logs and determine what is misbehaving.

Also, I'm going to declare that establishing mandatory access controls around any daemon is a modern and expected best practice. Please note that this does not excuse leaving the vulnerability unpatched. Defense-in-depth is to help catch flaws, not to shift the burden completely. Also, be very mindful of the idea that your server may be serving up compromises to visitors of your site.

Answer 6

4 exe files on my server were replaced with renamed versions of the same virus...My web server is running Ubuntu 10.4.3

Implies that you've already got file upload functionality on your server - which would be a good place to start looking for holes.

I know that server, once compromised, is pretty much "gone" forever

Not so. If you make adequate preparations, then its always possible to get the system back to a known state (host based IDS and backups). But even then it doesn't eliminate the original vulnerability. But it would eliminate any additional back doors installed.

including but not limited to WordPress, vBulletin, and several homemade products

OMG. Just because someone else wrote it doesn't mean it is secure. Just because it is open source doesn't mean it is secure. Even if you've got the skills to audit the code properly the Wordpress code base is huge, and vbulletin is big too. This will be a big task. Also both Wordpress and vbulleting (IRC) support third party plugins - which are often of much poorer quality than the core products.

If you were running apache, then I'd suggest using mod_security to try to identify the attack vector. But even at a fairly high level of load you should be able to reconcile the mtime on the file with the access log (as you've discovered, looking at file names is a waste of time).