I have a PHP sheller file (like c99 file) but it's encoded with this functions:
The clamav virus scan can't detect it. How can I detect a virus and sheller like that?
Information Security Stack Exchange is a question and answer site for information security professionals. It only takes a minute to sign up.Sign up to join this community
ClamAV is possibly not the correct product to identify php shells. This is because there are endless ways to encode the shell, and it is not tailored to understand PHP code.
There are other technologies that are more tailored to detect malicious php code. Emphosha is one alternative that understand the php preprocessor, and can raise warning based on functions considered used for obfuscating code + able to detect known shells.
Signature based malware detectors can only be as good as the database of signatures they have available. This makes for a great business model for vendors of such products - the database will always need updating to support new malware. Writing heuristic based malware scanners is really hard - the ones I've looked at in any depth do provide additional coverage to zero day exploits, but not much.
An alternative approach to trying to find bad stuff is to keep track of the good stuff using a host based IDS. This has to be embedded in your deployment process. Further, it cannot be applied to artefacts considered as data - consider CVE-2014-8449 or CVE-2016-8332; most services perform some sort of transformation or storage of data, hence are designed to ingest data. The IDS cannot validate content it has not been told is good.
An extension to the concept of tracking file signatures is running a trusted platform - where all executables must have their signature checked before being run, however this is computationally expensive, more expensive in effort for maintenance and the main players in this market treat it as a mechanism for protecting their monopoly - and often fail completely to extend coverage to artefacts considered as data.
A further consideration is that any inline component (and some offline ones) increases your attack surface. Sadly, just because it says "Security" on the box, any bolt on product may add vulnerabilities to your system.
The most rigourous solution is to use a combination of signature / heuristic scanners along with an IDS.