Automatic detection of a compromised web server

Question

I'm trying to figure out if it is possible to automatically detect the the fact that the HTML or javascript files on a server have been tampered with or changed by an attacker.

Our website, say www.example.com/index.html, when loaded in a browser, also loads https://www.example.com/scripts/example.min.js inside a <script></script> tag. If someone were to somehow hack into my server and swap out the example.min.js file with a modified one, is there any way I can automatically detect said intrusion?

One way to do it would be to run a program on an independent secure server that queried https://example.com every few minutes and compared the SHA of the index.html and example.min.js file to the last known good values.

Question 1:- assuming the polling interval is acceptable, is this a strong enough defense? Could the malicious code fool the polling code?

Question 2:- Is there a better way to reduce the window of risk other than reducing the polling interval, which creates unnecessary traffic.

Answer 1

There are a number of local processes that will watch files and directories for any changes, writes, deletions, and accesses. When these events occur, the process creates an event log through syslog. This can happen in a second.

If the syslog entries are sent to a remote server (as they should be) you will have nearly instant alerting to file changes, without the possibility of "fooling" anything, without any "unnecessary" web traffic.

There are pre-packaged software to do this, or it can also be done using custom scripts (pre-packaged, tested software tends to be better).

Answer 2

Question 1:- assuming the polling interval is acceptable, is this a strong enough defense? Could the malicious code fool the polling code?

No, its not a strong enough defense. Yes, the malicious code can identify the polling server/service by its IP, User Agent etc. and serve the clean file to your polling agent while continue to serve malicious files to others. Sure, you can try to prevent detection by changing IPs and User Agents of your agent regularly, but so can the malicious code adapt to detect your change.

While serving different content based on client is it in itself a trivial thing to do, the fact that the attacker knows that such a safeguard is in place is non-trivial (its more like a security through obscurity). The way an attacker can know about such a safeguard is when he knows your architecture (internal threat) or if you go through the compromise->fix->compromise cycles enough times for him to narrow down the cause of compromise detection.

Question 2:- Is there a better way to reduce the window of risk other than reducing the polling interval, which creates unnecessary traffic.

The better way to reduce the risk is to have defense in depth:

• Have host based file integrity tools like tripwire to watch the files in question.
  • Defeating this defense will require host compromise
• Depending on how sensitive the files are and your paranoia level, configure your deployment tools to rebuild your files every n hours.
  • Defeating this defense will require a network compromise.
• Remote monitoring of your files as you described.
  • Defeating this defense will require 1.) either compromise of remote machine or 2.) enough knowledge/reconnaissance of your system.
• Deploy host based intrusion detection/prevention tools like snort, fail2ban, denyhosts to detect intrusion.
  • Defeating this defense will require an attack which doesn't create a lot of noise in log files and/or evades signature based detection eg. application layer flaws.
• Deploy a web application firewall eg. ModSecurity to detect application level anomalies.
  • Defeating this will require an lower level attack which should ideally get detected in the aforementioned point.

Even having all the measures above cannot guarantee security, however they reduce a lot of attack surface and make a successful and undetected compromise difficult.

Answer 3

As far as Question 1 goes, it would be easy for malicious code to fool the polling code. As am example, a lot of PHP malware looks at User Agent strings, and gives back 404 codes to Google, Bing, Yahoo and others. It would be almost as easy to keep track of IP addresses that make repetitive requests, and spoof them. Given a dedicated opponent, the whole thing would devolve into an arms race - who can write the most subtle tests for authenticity vs who can spoof subtle tests for authenticity. That's not to say that such an arms race would occur, or that you personally can't win such an arms race, but rather, in general, malicious code and fool testing code.

I think your problem is very closely related to the difficulty of merely monitoring a service over the network. Even something like a remote procedure call or REST interface can be difficult to monitor correctly in the face of bizarre enough bugs. Having tried this in the past, I can assure you that you end up in an "arms race" against weirder and weirder bugs.