Anomaly-based Malware Detection in Web Applications

Question

I am partly responsible for the security and malware detection (in general) of a huge number of hosting accounts. Our methods rely heavily on signature-based detection provided by ClamAV for which we have also created a pretty decent signature database ourselves (500+ samples).

We receive up to 30-40 notifications on a daily basis for malicious code being spotted in a user's account. As time flows, hackers get more intuitive and always manage to keep up with our (whitehat community as a whole) progress as we attempt to cease their attempts of exploiting vulnerabilities. However, it appears that signature-based detection of malicious code is increasingly becoming insufficient - we keep finding new malware samples that our scan has failed to detect initially. Therefore I took the liberty to attempt and design a system for strict anomaly detection.

I am asking for advice whether my current plannings can be considered reasonable since this is my first time taking up on such a task.

I think the more conditions I check the file against, the better results I can get (eliminating false positives as much as possible). So far I have created the following list of conditions:

• file location (5%)
• filename (10%)
• code tidiness/alignment (5%)
• repeating patterns (5%)
• weird/non-logical names of data structures (10%)
• stacking of multiple functions (3%)
• error reporting disabled (5%)
• error suppression used (2%)
• contains hex (i.e. \x73) (3%)

Where each of the conditions gives the file a certain percentage of it being potentially malicious or has malicious content within. My questions are:

1. Is this a good/right approach to developing a strict anomaly-based detection system? If not, why?
2. Can you give me example(s) of what else I can check a file against? Any tips or any useful information?

Do consider that:

1. It is targeted at web applications and mostly PHP files;
2. A lot of our clients use CMS's. Maybe it is better to base the system on a certain CMS?

Thank you.

Answer 1

I'm currently doing some research in the area of network security, where my topic is anomaly detection for log messages. However, I hope that I will be able to provide some insights for your problem.

(1) Signatures are ugly, but they do work. Although everybody is talking about anomaly detection and similar topics, nowadays signatures remain the most reliable way to detect malicious code/activity/etc. So you should continue using them and do not hope, that some anomaly detection will be able to fully replace it.

(2) The signatures are, of course, very limited and they cannot identify novel type of malicious code/attacks/etc. Then there are attempts to solve this problem with anomaly detection. The main disadvantage of anomaly detection is that it always produces false positive alerts that you should process and filter out. The current trend is to build a hybrid systems: first apply signatures and maybe some queries, and then perform anomaly detection in addition to it.

(3) It should definitely be possible to create an anomaly detection approach for your problem based on some machine learning techniques (based on your list of conditions; theoretically, it can be also possible to analyse PHP code i.e. perform static code anomaly detection, but this could be rather complicated/big project). However, I won't expect that it will catch all malicious code. The realistic number could be something like 70-80% detection rate with 10-20% false positive rate. So if you check 1000 files and 10 of them are malicious, probably in the best case you will be able to develop a system that will return you 100 alerts, where 8 of 10 malicious files will be detected, but the rest (let's say 92 other alerts) will be false positive. Also, probably 6-7 of these 8 you can have already detected with signatures and queries.

I hope that I'm not too pessimistic here. Sometimes, for specific use cases, it is possible to achieve 98-99% true positive rate with 1-2% false positive rate, but I'm not sure if you will have so much luck in your case. Also I do not know enough details about your use case to make any predictions.

(4) However, if you will be able to rank your results, then the top of the anomaly detection output will contain more true positive and less false positive alerts. For example, if your system will return 100 ranked alerts, you can only look on top 30 of them. And these 30 will contain 6 true positive results and only 24 false positive (which is to some extend better than 8 true positive and 92 false positive). Next, if you look at 5 most highly ranked alerts, they will probably contain 1-2 true positive and 3-4 false positive. And this is what you can check on the daily basis in addition to your signature-based detection results.

You can hire a data mining/machine learning expert to build such a system for you, or try to do it yourself, if you have enough skills.

I have not found too much relevant research papers (which is not a good sign), but I hope that the following can be relatively useful for you:

• Learning to Detect Malicious Executables in the Wild. However, this should be a misuse detection, not an anomaly detection.
• A static heuristic approach to detecting malware targets

Finally, there should be some alternative solutions to this problem. I have googled for it and here is what I have found so far:

• PHP malware finder

Also not too much (just one link), but I hope you can try to search more, there should be some techniques like sandboxing. Maybe you can check this pdf for relevant info.

Answer 2

This isn't meant as a complete answer to your question, but it's longer than what fits in a comment, so here goes:

I think that's an interesting approach, as long as you're aware that it will obviously never be able to catch all malware, because no matter which criteria you set up, they can be worked around.

For example, the last one, "contains hex", can be easily be worked around by specifying the hex values as an array of decimal numbers, or as a base64-encoded string, and "code tidyness" can be automatically achieved by running a file through xxxtidy, where xxx stands for your favorite programming language's version of a code tidier.

I'm also not sure how you plan to detect "weird, non-logical names of data structures". That seems like an impossible problem to me. And I'm not sure what "stacking of multiple functions" means; if I understand that correctly, that's something that will generate a huge number of false positives.

Still, the reason I find this approach interesting is because even though it will never be perfect, it probably will help in detecting suspicious programs, just like jslint, pylint etc help spot problems in regular code.

If you know the programming language you're dealing with, you could also check for suspicious calls like calls that try to start a system shell, calls that try to open network sockets, redirect input or output, calls to eval, calls that try to open external files for reading or writing, etc.

Implementation

This is where I think you'll run into difficulties, at least if you're basing your idea on scanning individual php files. I wouldn't want to write parsers that can reliably detect all the mentioned criteria. That sounds like an awful lot of work.

Alternative

An alternative to looking for specific code patterns in php files might be to look for specific consequences of such code patterns being executed. For example, if you're looking for code that tries to open a network socket, that's probably fairly hard to detect, but periodically running nmap or netstat and checking for ports that shouldn't be open is easy. Checking for code that writes to external files is difficult, but building a list of, say, all executable files under /tmp, or files with recent timestamp under /usr/local/bin, is easy. You could also look for active root shell processes, and so on.

Prior art

There are various programs which already do parts of that. For example, rkhunter under linux looks for suspicious changes to your file system, tripwire and snort would be two other solutions to check out.

A quick googling session also turns up https://github.com/nbs-system/php-malware-finder and https://www.decalage.info/python/balbuzard, which both seem to do exactly what you're proposing.