I am partly responsible for the security and malware detection (in general) of a huge number of hosting accounts. Our methods rely heavily on signature-based detection provided by ClamAV for which we have also created a pretty decent signature database ourselves (500+ samples).
We receive up to 30-40 notifications on a daily basis for malicious code being spotted in a user's account. As time flows, hackers get more intuitive and always manage to keep up with our (whitehat community as a whole) progress as we attempt to cease their attempts of exploiting vulnerabilities. However, it appears that signature-based detection of malicious code is increasingly becoming insufficient - we keep finding new malware samples that our scan has failed to detect initially. Therefore I took the liberty to attempt and design a system for strict anomaly detection.
I am asking for advice whether my current plannings can be considered reasonable since this is my first time taking up on such a task.
I think the more conditions I check the file against, the better results I can get (eliminating false positives as much as possible). So far I have created the following list of conditions:
- file location (5%)
- filename (10%)
- code tidiness/alignment (5%)
- repeating patterns (5%)
- weird/non-logical names of data structures (10%)
- stacking of multiple functions (3%)
- error reporting disabled (5%)
- error suppression used (2%)
- contains hex (i.e. \x73) (3%)
Where each of the conditions gives the file a certain percentage of it being potentially malicious or has malicious content within. My questions are:
- Is this a good/right approach to developing a strict anomaly-based detection system? If not, why?
- Can you give me example(s) of what else I can check a file against? Any tips or any useful information?
Do consider that:
- It is targeted at web applications and mostly PHP files;
- A lot of our clients use CMS's. Maybe it is better to base the system on a certain CMS?