How do you know your server has been compromised?

Question

I recently helped a client who had their server hacked. The hackers added some PHP code into the header of the homepage redirecting the user to a porn website — but only if they came from Google. This made it slightly harder for the client to spot. The client would see the website fine. Only new website visitors from Google would be directed to the porn site.

Last night a similar thing appeared to happen to a different client. I assumed it was a similar hack, but when I checked the codebase I could not find any malicious code. His chrome browser is redirecting from the clients website to www(dot)pc-site(dot)com. I cannot replicate this behaviour. I guess it is possible that malicious code is being added and removed. So I need a more comprehensive way to tell if the server has been hacked.

Only 2 developers have access to this dedicated server (and the hosting company Rackspace). The server is Red Hat Linux.

What are the steps I go through to find out if the server has been hacked?

Answer 1

UPDATED

I would check the following:

1. Logs. If you have root access you should check things like history which will give you command history and log files in /var/logs.

2. Baseline. If you have a baseline like file hashes to work with for application and system files this will help a lot. You can also use backups to compare a previous state. If using a backup to compare files, use a slightly older one if you can. The site may have been compromised a while before and it is only now that the redirect has been activated.

3. Check any includes. The files may not be on your server. They may be script includes such as <script src=”http://baddomain.com/s.js” /> or iframe type tags. Also do not exclude images, PDFs of Flash (SWF), video files. It is a fairly common trick to embed links in to files of a different content type. I would suggest you inspect them by hand particularly at the start and end of a file. The file may be completely a link/html/javascript or may be a legitimate image file with a link trailing at the end of the file.

4. Check for unusual file dates, sizes and permissions e.g. 777.

5. Check cron jobs for unusual jobs. Someone compromising a system will often leave a back door to get back in again and again. Cron is a very popular way to do this if they managed to get that far.

6. Check for the absence of files, you may not be able to have access to logs but the absence of them is equally a tell tail sign that someone has cleaned up after themself.

7. Use search engines. Not surprising search engines are great at finding everything. Use directives like site: e.g. site:yoursitehere.com baddomain.com see if you get any hits.

8. Often a link or redirect will be obfuscated so long javascript code with single letter variables should be analyzed carefully.

9. Do a packet capture with a tool like Wireshark or tcpdump from a secure workstation to the site. Save it to file and search the file for a parts of the url.

10. Check database records that may be queried or updated. The link could be injected in the database not the PHP.

11. Don't exclude the client's workstation. Use a free online virus scanner if need be. Also check nslookup and see what that resolves to. Check browser extensions, clear cache and check hosts files.

To clean it up (if you are compromised) you really do need to go back to bare metal and reinstall. It is painful but is really the only way to be sure that you have got the whole lot.

To prevent it in the future you should be doing the following (although you may already be doing some of these):

1. Harden servers, including using vendor recommendations on secure configurations, using up-to-date software. Apply tight security control such as permissions, password policies. Also see folder and file permission shared host advice.

2. Implement quality control proceedures such as testing on low security environments, code review and testing.

3. Have your web application / web site vulnerability tested by a professional certified tester at least once. Look for EC-Council, ISO 27001 and PCI certified testers. http://www.eccouncil.org/certification/licensed_penetration_tester.aspx

4. Check out OWASP www.owasp.org and http://phpsec.org/projects/guide/2.html for web application security resources.

5. Use Intrusion Prevention System (IPS) tools. However depending on your hosting provider you may have limitations on what you can use. Host based IPS tools should be ok if you have a dedicated virtual machine.

Hope that helps. Otherwise maybe you could provide more information about the systems you are running?

Answer 2

As @Dgarcia said, a quick method is to use something like Tripwire or other tool which monitors files or the hashes of files to check for changes. This works to identify servers compromised by many types of attack.

1. It may not work for ones where a rootkit has been installed that counteracts this process.
2. It will not work for servers which have fallen prey to a memory-only compromise or one that doesn't touch the files you are monitoring.

For 1, your only option is a rebuild from scratch

For 2, your best option is a rebuild from scratch, as any compromise could implement backdoors that will break anything you try to fix, but other steps could be useful:

• check your webserver and php versions and use these to search on an Advisories list for known exploits - this will help you identify areas which may have been compromised. Then
• check your web application code
• check your webserver configs
• check the client's machine (for hosts file, DNS etc) as it may actually be the problem

Answer 3

This is a tough question to answer because it is so broad. There are two categories of "hacks" in my book - minor and serious. I would class a rootkit in the serious category and your average script injection attack as minor. While with minor attacks you can clean them up, you can't be 100% certain you've removed them or closed all access to repeat the attack but you can be 99%-ish certain by analyzing the attack for key factors such as "Was this person a good programmer?" and "What was the person's intent?" Rootkits are nasty business. Removing a rootkit requires a complete wipe and restore. Detecting one remotely is nearly impossible - you have to have physical access to the machine and a boot disk in hand to be certain.

More importantly is prevention. The adage "an ounce of prevention is worth a pound of cure" is completely true in this context. Install software that lets you monitor various aspects of the system and sends daily or even hourly reports. Tripwire was mentioned, but there are other tools out there too. I recommend using a couple of different tools - homegrown ones are harder to locate and aren't difficult to author. You want to build a solid defense and limit access to the system. Don't just let anyone in the world have access to the SSH port (at least restrict it by IP address/small range of IPs). Stick a dedicated firewall in front of each server so there is an extra layer of protection. You don't want to let the box itself be the only line of defense. Only manage critical data with the server over SSH/SSL so everything is encrypted and free from prying eyes. Don't ever manage your servers from open WiFi networks.

A lot of sites use MySQL or a similar database. Detecting things like XSS attacks or other rogue data in a database isn't easy because there are schema-dependent issues. I haven't seen any solutions for this problem but I wouldn't doubt they exist.

Answer 4

A fast method is to have the md5 of the all the files that you know are healthy. If you suspect your site is behaving bad or as a regular inspection you can do the check over the files. If any of the md5 doesn't match you can diff the files and peruse the changes.

Obviously this does not work with dinamic files: logs, database dumps, etc. If you can't track the changes.

There are, of course, multiple methods (inspect logs...) and preventions, but this is an easy and fast way.