Take the 2-minute tour ×
Information Security Stack Exchange is a question and answer site for Information security professionals. It's 100% free, no registration required.

I need some help tracing a vulnerability on my server. For the second time, my server has been compromised with files being replaced with virus-ridden downloads.

According to the filesystem dates, over a period of 45 minutes 4 exe files on my server were replaced with renamed versions of the same virus.

My web server is running Ubuntu 10.4.3 LTS with kernel version 2.6.32-31-generic, kept completely patched and up to date.

The only way of accessing the shell is via SSH and a password-protected private key that I have with me on a USB stick. Password SSH login is disabled and the server logs (which I know can be modified, but I have good reason to believe they haven't) indicate that SSH was not used to log into the server.

The web serving software stack is very complicated. There's PHP (5.3.3-1) w/ Suhosin v0.9.29, nginx 1.0.9 (now updated to 1.0.10), Tomcat (in a jail and I suspect not associated), and MySQL 5.1.41.

I admit that at the time of the first attack, I had been content to blithely chmod -R 777 my web directory for headache-mitigation purposes. Now I run a complete mess of PHP scripts including but not limited to WordPress, vBulletin, and several homemade products; the first two of which are always up to date and the latter has been written with fairly great care to escape or normalize any user-inputted values.

Given the weak file permissions but strongly-locked down server access, I was highly tempted to suspect a vulnerability in one of the many PHP scripts that allowed the execution of random code.

I have since completely locked down the file permissions. nginx/php both run as www-data:www-data with all files given only execute and read permissions (chmod -R 550 /var/www).

Yet today, after all this, my server was again compromised.

The thing is, the files that were replaced still have 550 permissions, the SSH logs indicate no log in, and I'm completely lost as to what to do or try next.

I tried to recreate the attack on the paths that were replaced with a very basic PHP script:

$file = fopen('/var/www/mysite.com/path/to/file', 'w');
fwrite($file, 'test');
fclose($file)

But that gave me the appropriate permissions denied error.

Can anyone please, please advise me where to look next for the source of this vulnerability? Am I missing something with my file permissions?

I know that server, once compromised, is pretty much "gone" forever. But that's not really an option here. I've recursively grepped my entire /var/log folder for the afflicted file names hoping to find something, but nothing came up.

I also searched for any scripts in the cron folder or elsewhere that might have been placed at the time of the first attack to attack again at a later date, but (a) found nothing, and (b) shouldn't find anything as the files in /etc/ are not modifiable by www-data (assuming a nginx/PHP point of infiltration).

I should add that both times I have grep'd the nginx access logs (combined style) for the names of the infected files, but found nothing. I do understand/realize that many ways of obscuring the file names from my greps exist, however.

share|improve this question
6  
As you've said, once a server is compromised it should not be trusted again. Without a full system re-build, you cannot easily be certain that the attacker hasn't left behind a rootkit that is preventing you from detecting his resident back-doors. And, of course, the use of his back-doors won't show up in your SSH logs. –  Iszi Nov 28 '11 at 6:50
3  
It seems to me that what Mahmoud is trying to do here is find out how the attacker got in so he can fix the problem. Re-building the system from scratch will just result in it being compromised again in exactly the same way. –  Ladadadada Nov 28 '11 at 18:45
2  
No, re-building the system from scratch will return it to a known-trustworthy state so that normal operations can resume. Preferably, this should be done on separate hardware. That way, if you're really worried about where the attack came from and how, you can perform analysis on the old hardware. –  Iszi Nov 28 '11 at 21:24
3  
@MahmoudAl-Qudsi - by definition, you can not treat your system as clean right now. Do you have any sensitive information on there - passwords, bank account details, customer information, intellectual property? Finding the attack vector should always come second to ensuring the safety of customer information or your sensitive information. –  Rory Alsop Nov 29 '11 at 8:35
2  
@RoryAlsop Has hit the nail right on the head - the exact point I was trying to express. A compromised "distribution server" still puts you and your customers at high risk because it could be used to distribute malware to its end-users' systems. Regardless of what the server's role is though, it should be taken offline until it can be trusted again. Otherwise, if the attacker still has a back-door into it, it could be used as a jumping-off point to facilitate attacks on your more critical and sensitive systems. –  Iszi Nov 29 '11 at 18:25

7 Answers 7

up vote 11 down vote accepted

Some techniques for trying to find how your attacker got in:

  1. Look at the timestamps on any files you know the attacker changed then look through all your logs for entries as close to each timestamp as possible. As others have said, the web access logs and web error logs are the most likely to hold the evidence of the original attack vector but other log files may hold clues as well. Error logs often hold the best clues. The logs of all network-accessible daemons are also good places to look.
  2. It may also be worthwhile looking for other files with timestamps close to the ones you know. /etc/passwd is an obvious one but even log files can be suspicious if they have an unusual timestamp. If logrotate runs at the same time every day and one of your log files has a timestamp that doesn't match this time, it was probably altered to cover his tracks, and now you know a little more about what he did. Don't forget the .bash_history files in user's home directories. the find command should be able to handle this for you.
  3. Run scalp over your web access logs. The original attack may have happened some time before the files appeared. Scalp will produce false positives but it should narrow down the potential suspect log entries for you to manually analyse for irregularities. It may also miss the attack entirely - it's not perfect - but it may help.

Don't spend too much time on forensics in the compromised system. As others have noted, the attacker has had the opportunity to remove all evidence of the original attack and to add a rootkit to hide his continued presence. If he has missed something or if he has not even tried, the above tasks may work but if he has hidden himself well then you would be just wasting valuable time.

If you fail to find the source of the attack, wipe and re-install the server in question but with some additions in order to catch him next time.

  1. Ship your logs off to a different server using some variant of syslog. (rsyslog and syslog-ng are the recommended choices) Preferably, this server should do nothing but receive logs and should not share login keys or passwords with any other server. The goal is to make sure your logs cannot be tampered with or deleted.
  2. Add extra logging beyond the default. Jeff already mentioned AppArmor and since you are using Ubuntu this will probably be the best choice. Make sure its logs are sent to the logging box.
  3. Install and turn on the audit daemon. Make sure its logs are sent to the logging box.
  4. Run an IDS such as Snort, PHP-IDS or mod_security. (or more than one, these tool don't all do exactly the same job) Some hardware firewalls come with IDS/IPS modules. Make sure the logs from the IDS are sent to the logging box.
  5. Add a file integrity monitoring system such as AIDE or Tripwire. Make sure the logs from these tools are sent to the logging box.
  6. Monitor your logs. Falling short of a commercial SIEM system, something like Splunk can be installed for free and can analyse a limited quantity of logs. Set up rules to match what is normal for your servers and filter them out. Whatever is left is worthy of closer inspection.

There is much more you can do if you have the time and the money, such as full network packet captures, but realistically this is about all a lone sysadmin can be expected to handle. If the attacker shows up again, you will be much more likely to find the attack vector and much more likely to detect him as soon as the attack is made.

share|improve this answer
    
Thank you for this very comprehensive post. I really do get the feeling that the attacker was nowhere near as talented as they come, the first attack was botched and this second left many telltale signs.... which makes it all the more frustrating that I'm unable to find the attack vector yet. I'm running scalp now, we'll see how it goes. –  Mahmoud Al-Qudsi Nov 29 '11 at 4:56
2  
Somehow I hadn't come across OSSEC before Bassec mentioned it. It looks to do both AIDE's and Splunk's jobs. Probably worth investigating. –  Ladadadada Nov 29 '11 at 11:18
    
+1 for Splunk. Please note that the free version supports up to 500MB of logs. If you are doing more it won't be much of a help. Enable rsyslog (if you did not already) to ship the logs to a different server. Depending on your need you can set it up to rotate every 5m, 1h, etc. Please note that some of the suggestions will add penalty to performance. Which under some circumstances can inflect DoS on your server. The good thing about OSSEC is that you can deploy the agent and have the server somewhere else. You can still monitor your system (failed logins, etc) without affecting performance. –  Bassec Nov 29 '11 at 17:57

@Iszi is absolutely correct here. You need to completely wipe and rebuild, as a good rootkit will prevent you seeing any evidence of its existence.

Otherwise there is a strong likelihood that anything you do now is pointless. In any case, you can no longer trust the server.

share|improve this answer

Just to add a couple of thoughts to the comments already provided. As @symcbean says, third party code is a likely source of issues.

If I was to hazard a guess, I'd look at the Wordpress application and associated plugins as being a likely source of your compromise. There have been a large number of security issues discovered in Wordpress plugins recently and many of them are being actively exploited, so if you have that installed and not 100% up to date you could well have issues.

In terms of addressing the Wordpress piece of your problem, you could look at wpscan which is a wordpress security scanner and also the secure wordpress plugin

share|improve this answer
    
+1 I spent a couple of hours last week looking at a couple of random WP plugins and found several bugs, including 2 XSS and a directory traversal vuln in one plugin. –  bstpierre Nov 28 '11 at 12:25
    
Rory, thanks for your answer. I'm running those now. But my question is this: how can it be PHP when my intentional reproduction of the attack failed due to file permissions? –  Mahmoud Al-Qudsi Nov 28 '11 at 19:27
    
WordPress is fine. I have minimal plugins, wpsecure only found a FPD in WP core (the least of my worries at the moment), and a warning about an SQL injection in an older version of one my plugins (which I'm not running). –  Mahmoud Al-Qudsi Nov 28 '11 at 19:42

Muhammad, in some cases I saw the attack was conducted by a bot that exploited a vulnerability on the system (Usually PhpMyAdmin) and uploaded files to the server (mainly /tmp). What happens is that those files stay there and admin won't notice until the attacker go over their bot logs and start building on the first compromised.

It is quite possible that a vulnerability in PhpMyAdmin or vBulletin or some script you are using was exploited. Later on you updated the vulnerable app, but the compromise already happened.

If you don't do a rebuild, what happens is that files left by the attacker were used to compromise your system, again. Installing OSSEC on your system and monitoring critical files/folders will help you detecting such activity and allows you to take action more quickly.

share|improve this answer
2  
+1 for OSSEC, it's definitely worth running on production boxes to keep an eye out for odd occurrences –  Rоry McCune Nov 29 '11 at 11:29

4 exe files on my server were replaced with renamed versions of the same virus...My web server is running Ubuntu 10.4.3

Implies that you've already got file upload functionality on your server - which would be a good place to start looking for holes.

I know that server, once compromised, is pretty much "gone" forever

Not so. If you make adequate preparations, then its always possible to get the system back to a known state (host based IDS and backups). But even then it doesn't eliminate the original vulnerability. But it would eliminate any additional back doors installed.

including but not limited to WordPress, vBulletin, and several homemade products

OMG. Just because someone else wrote it doesn't mean it is secure. Just because it is open source doesn't mean it is secure. Even if you've got the skills to audit the code properly the Wordpress code base is huge, and vbulletin is big too. This will be a big task. Also both Wordpress and vbulleting (IRC) support third party plugins - which are often of much poorer quality than the core products.

If you were running apache, then I'd suggest using mod_security to try to identify the attack vector. But even at a fairly high level of load you should be able to reconcile the mtime on the file with the access log (as you've discovered, looking at file names is a waste of time).

share|improve this answer
    
No. The exes were manually place by me via scp. And I know for a fact that open source and 3rd party software isn't the best, and I actually strongly suspect vB. –  Mahmoud Al-Qudsi Nov 28 '11 at 19:15
    
I should mention that I manually viewed the access logs searching for the intrusion timestamp, but I was unable to find anything. –  Mahmoud Al-Qudsi Nov 28 '11 at 19:39

Usually when people say that they can't afford to wipe a server, they mean that the time to reset the configuration will be too much. However, there's a way to go about this and not have to find the vulnerability immediately -- though you must replace compromised files. I suggest implementing security before exposing your clean files to the Internet. What broke them once will break them again.

Steps below are Debian-focused (you didn't say which you had, so I took my own preference), but you can perform them on Red Hat systems as well. On a Red Hat based system, you may have an easier time substituting AppArmor suggestions with corresponding SELinux setup. Change "dpkg" to matching "yum" commands.

  • Build a new system using the packages you already have installed. dpkg --get-selections will give you a list that can be piped into the new system using dpkg --set-selections.

  • Configure apparmor to heavily restrict nginx and any other daemons (Tomcat, etc.) used by it. Allow reading only of configuration and web page files. Some writing might need to be enabled to the web directory depending on what you're doing. Make sure that your configuration forces any child processes to inherit the Apache restrictions.

  • Copy everything else back.

Pay very close attention to your AppArmor logs. When you see Apache try to violate its permissions, you can correlate that with your HTTP access / error logs and determine what is misbehaving.

Also, I'm going to declare that establishing mandatory access controls around any daemon is a modern and expected best practice. Please note that this does not excuse leaving the vulnerability unpatched. Defense-in-depth is to help catch flaws, not to shift the burden completely. Also, be very mindful of the idea that your server may be serving up compromises to visitors of your site.

share|improve this answer
    
Thank you for your advice. I will read up on AppArmor. What do you mean by establishing mandatory access controls? Are you referring to ACL file permissions? They've all now been set... though it's probably too late. I am on Ubuntu server, it's mentioned in my post. –  Mahmoud Al-Qudsi Nov 28 '11 at 19:16
    
See en.wikipedia.org/wiki/Mandatory_access_control -- I'm referring to controls that are not set on the filesystem, but above it (no ACL, no chmod, etc.). wiki.ubuntu.com/AppArmor#Example_profile is an example for tcpdump that only allows it to write to a home directory and does not permit anything beginning with a "." or "/bin". That means even a compromised tcpdump app running SUID root couldn't stomp all over your configs. –  Jeff Ferland Nov 28 '11 at 20:22
    
I've been studying apparmor and just created profiles for nginx and php, so I now understand what you're referring to. –  Mahmoud Al-Qudsi Nov 28 '11 at 20:57

The first suspect would be the plugins for wordpress/vb that you might have installed, those could really be bad for security, sometimes there could be an upgrade to wordpress or the plugin only, and might look as if its working fine, but they could introduce serious security issues.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.