I am partly responsible for the security and malware detection (in general) of a huge number of hosting accounts. Our methods rely heavily on signature-based detection provided by ClamAV for which we have also created a pretty decent signature database ourselves (500+ samples).

We receive up to 30-40 notifications on a daily basis for malicious code being spotted in a user's account. As time flows, hackers get more intuitive and always manage to keep up with our (whitehat community as a whole) progress as we attempt to cease their attempts of exploiting vulnerabilities. However, it appears that signature-based detection of malicious code is increasingly becoming insufficient - we keep finding new malware samples that our scan has failed to detect initially. Therefore I took the liberty to attempt and design a system for strict anomaly detection.

I am asking for advice whether my current plannings can be considered reasonable since this is my first time taking up on such a task.

I think the more conditions I check the file against, the better results I can get (eliminating false positives as much as possible). So far I have created the following list of conditions:

  • file location (5%)
  • filename (10%)
  • code tidiness/alignment (5%)
  • repeating patterns (5%)
  • weird/non-logical names of data structures (10%)
  • stacking of multiple functions (3%)
  • error reporting disabled (5%)
  • error suppression used (2%)
  • contains hex (i.e. \x73) (3%)

Where each of the conditions gives the file a certain percentage of it being potentially malicious or has malicious content within. My questions are:

  1. Is this a good/right approach to developing a strict anomaly-based detection system? If not, why?
  2. Can you give me example(s) of what else I can check a file against? Any tips or any useful information?

Do consider that:

  1. It is targeted at web applications and mostly PHP files;
  2. A lot of our clients use CMS's. Maybe it is better to base the system on a certain CMS?

Thank you.

  • 1
    I do not envy your job... I wouldn't even know where to start. It is so hard to do security when you give them the ability to upload literally anything
    – MikeSchem
    Mar 3, 2017 at 0:23
  • 1
    the list of conditions in your question reminded me of features used in machine learning algorithms designed to detect spam. Its a classification problem. Maybe this article will help you: Detecting Malware Pre-execution with Static Analysis and Machine Learning. The article discusses feature selection, model selection and tuning
    – julian
    Mar 3, 2017 at 2:04
  • Is there a way you could prevent the malicious code from doing damage instead of trying to detect it? Run everyone's account under its own VM with restricted network access, and allow them to punch holes through the firewall on a case by case basis. This would make most web malware useless as it can no longer spam or contact its C&C server, thus making it less useful for attackers to try and crack the sites. Mar 3, 2017 at 11:56
  • Out of curiosity, have you taken a look at some WAF solutions, like ModSecurity or Naxsi? It is still signature based, but one advantage may be the ability to establish a profile for normal use ahead of time, from which ModSecurity at least can generally detect deviations. I am not totally clear on what users are uploading, but is your concern protecting your systems (and if so, what is the vector from upload to compromising a system), or your users/reputation (i.e. avoid hosting malware)? For the former, maybe Cuckoo, or even a canary of some form, would help Mar 3, 2017 at 13:40
  • @iwaseatenbyagrue, We have mod_security running already and we have written customer rules for it as well. It helped reduce the attacks by a huge percentage. I am currently researching means of detecting malware once it has been uploaded in a user's account. I will check out Cuckoo, thanks!
    – McJohnson
    Mar 3, 2017 at 13:45

2 Answers 2


I'm currently doing some research in the area of network security, where my topic is anomaly detection for log messages. However, I hope that I will be able to provide some insights for your problem.

(1) Signatures are ugly, but they do work. Although everybody is talking about anomaly detection and similar topics, nowadays signatures remain the most reliable way to detect malicious code/activity/etc. So you should continue using them and do not hope, that some anomaly detection will be able to fully replace it.

(2) The signatures are, of course, very limited and they cannot identify novel type of malicious code/attacks/etc. Then there are attempts to solve this problem with anomaly detection. The main disadvantage of anomaly detection is that it always produces false positive alerts that you should process and filter out. The current trend is to build a hybrid systems: first apply signatures and maybe some queries, and then perform anomaly detection in addition to it.

(3) It should definitely be possible to create an anomaly detection approach for your problem based on some machine learning techniques (based on your list of conditions; theoretically, it can be also possible to analyse PHP code i.e. perform static code anomaly detection, but this could be rather complicated/big project). However, I won't expect that it will catch all malicious code. The realistic number could be something like 70-80% detection rate with 10-20% false positive rate. So if you check 1000 files and 10 of them are malicious, probably in the best case you will be able to develop a system that will return you 100 alerts, where 8 of 10 malicious files will be detected, but the rest (let's say 92 other alerts) will be false positive. Also, probably 6-7 of these 8 you can have already detected with signatures and queries.

I hope that I'm not too pessimistic here. Sometimes, for specific use cases, it is possible to achieve 98-99% true positive rate with 1-2% false positive rate, but I'm not sure if you will have so much luck in your case. Also I do not know enough details about your use case to make any predictions.

(4) However, if you will be able to rank your results, then the top of the anomaly detection output will contain more true positive and less false positive alerts. For example, if your system will return 100 ranked alerts, you can only look on top 30 of them. And these 30 will contain 6 true positive results and only 24 false positive (which is to some extend better than 8 true positive and 92 false positive). Next, if you look at 5 most highly ranked alerts, they will probably contain 1-2 true positive and 3-4 false positive. And this is what you can check on the daily basis in addition to your signature-based detection results.

You can hire a data mining/machine learning expert to build such a system for you, or try to do it yourself, if you have enough skills.

I have not found too much relevant research papers (which is not a good sign), but I hope that the following can be relatively useful for you:

Finally, there should be some alternative solutions to this problem. I have googled for it and here is what I have found so far:

Also not too much (just one link), but I hope you can try to search more, there should be some techniques like sandboxing. Maybe you can check this pdf for relevant info.

  • Thank you for your answer. Yes, I am aware that there will be a lot of false positive results but I hope that during the testing phase I can reduce those and increasing the criteria might also contribute to that. And yes, I do plan on running ClamAV with its signatures first and once it's done and I have removed the malware detected by it, I will run an anomaly detection.
    – McJohnson
    Mar 3, 2017 at 13:30
  • It's true that systems using artificial neural networks that are self-learning can produce quite good results but I am more inclined to define what normal usage of the system comprises using a strict mathematical model, and flag any deviation from this as an attack. By the way, the last paper you linked seems to contain a lot of useful information, thank you!
    – McJohnson
    Mar 3, 2017 at 13:30
  • I haven't meant neural networks here. Besides neural networks, there are a lot of other machine learning and data mining techniques such as SVM (or one-class SVM for anomaly detection) that exactly have "a strict mathematical model, and flag any deviation from this as an attack". Classical neural networks are not a best choice for anomaly detection, in this area usually ocSVM or other methods are applied. However, there are anomaly detection techniques based on neural networks: Self-Organising Maps and Replicator Neural Networks. Mar 3, 2017 at 13:49

This isn't meant as a complete answer to your question, but it's longer than what fits in a comment, so here goes:

I think that's an interesting approach, as long as you're aware that it will obviously never be able to catch all malware, because no matter which criteria you set up, they can be worked around.

For example, the last one, "contains hex", can be easily be worked around by specifying the hex values as an array of decimal numbers, or as a base64-encoded string, and "code tidyness" can be automatically achieved by running a file through xxxtidy, where xxx stands for your favorite programming language's version of a code tidier.

I'm also not sure how you plan to detect "weird, non-logical names of data structures". That seems like an impossible problem to me. And I'm not sure what "stacking of multiple functions" means; if I understand that correctly, that's something that will generate a huge number of false positives.

Still, the reason I find this approach interesting is because even though it will never be perfect, it probably will help in detecting suspicious programs, just like jslint, pylint etc help spot problems in regular code.

If you know the programming language you're dealing with, you could also check for suspicious calls like calls that try to start a system shell, calls that try to open network sockets, redirect input or output, calls to eval, calls that try to open external files for reading or writing, etc.


This is where I think you'll run into difficulties, at least if you're basing your idea on scanning individual php files. I wouldn't want to write parsers that can reliably detect all the mentioned criteria. That sounds like an awful lot of work.


An alternative to looking for specific code patterns in php files might be to look for specific consequences of such code patterns being executed. For example, if you're looking for code that tries to open a network socket, that's probably fairly hard to detect, but periodically running nmap or netstat and checking for ports that shouldn't be open is easy. Checking for code that writes to external files is difficult, but building a list of, say, all executable files under /tmp, or files with recent timestamp under /usr/local/bin, is easy. You could also look for active root shell processes, and so on.

Prior art

There are various programs which already do parts of that. For example, rkhunter under linux looks for suspicious changes to your file system, tripwire and snort would be two other solutions to check out.

A quick googling session also turns up https://github.com/nbs-system/php-malware-finder and https://www.decalage.info/python/balbuzard, which both seem to do exactly what you're proposing.

  • The tools you mentioned serve a bit of a different purpose than what I'm after. From what I read about them, they are based off patterns within the file only. I, on the other hand, want something more general/broad. Yes, the criteria you mentioned can easily be bypassed but that's why there will be so many conditions after all . It's natural for several of them to fail to recognize malware. I do not want to base mine on keywords, functions, calls etc because this is easily obfuscated and I cannot really go over every single obfuscation technique used out there.
    – McJohnson
    Mar 3, 2017 at 0:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.