I need some help tracing a vulnerability on my server. For the second time, my server has been compromised with files being replaced with virus-ridden downloads.
According to the filesystem dates, over a period of 45 minutes 4 exe files on my server were replaced with renamed versions of the same virus.
My web server is running Ubuntu 10.4.3 LTS with kernel version 2.6.32-31-generic, kept completely patched and up to date.
The only way of accessing the shell is via SSH and a password-protected private key that I have with me on a USB stick. Password SSH login is disabled and the server logs (which I know can be modified, but I have good reason to believe they haven't) indicate that SSH was not used to log into the server.
The web serving software stack is very complicated. There's PHP (5.3.3-1) w/ Suhosin v0.9.29, nginx 1.0.9 (now updated to 1.0.10), Tomcat (in a jail and I suspect not associated), and MySQL 5.1.41.
I admit that at the time of the first attack, I had been content to blithely chmod -R 777 my web directory for headache-mitigation purposes. Now I run a complete mess of PHP scripts including but not limited to WordPress, vBulletin, and several homemade products; the first two of which are always up to date and the latter has been written with fairly great care to escape or normalize any user-inputted values.
Given the weak file permissions but strongly-locked down server access, I was highly tempted to suspect a vulnerability in one of the many PHP scripts that allowed the execution of random code.
I have since completely locked down the file permissions. nginx/php both run as www-data:www-data with all files given only execute and read permissions (
chmod -R 550 /var/www).
Yet today, after all this, my server was again compromised.
The thing is, the files that were replaced still have
550 permissions, the SSH logs indicate no log in, and I'm completely lost as to what to do or try next.
I tried to recreate the attack on the paths that were replaced with a very basic PHP script:
$file = fopen('/var/www/mysite.com/path/to/file', 'w'); fwrite($file, 'test'); fclose($file)
But that gave me the appropriate permissions denied error.
Can anyone please, please advise me where to look next for the source of this vulnerability? Am I missing something with my file permissions?
I know that server, once compromised, is pretty much "gone" forever. But that's not really an option here. I've recursively grepped my entire /var/log folder for the afflicted file names hoping to find something, but nothing came up.
I also searched for any scripts in the cron folder or elsewhere that might have been placed at the time of the first attack to attack again at a later date, but (a) found nothing, and (b) shouldn't find anything as the files in /etc/ are not modifiable by www-data (assuming a nginx/PHP point of infiltration).
I should add that both times I have grep'd the nginx access logs (combined style) for the names of the infected files, but found nothing. I do understand/realize that many ways of obscuring the file names from my greps exist, however.