I've been reading a lot about polymorphic and metamorphic malware and how they evade signature-based detection once they have been discovered. These techniques seem to be utilized to evade recurrent detections by AVs and IDSs. It occurred to me that it might be a high priority goal of malware authors to prevent their malware from being ever being analyzed in the first place. Are there any analogous techniques that malware authors have employed in order to prevent their malware from being first discovered so that they never get reported/analyzed in the first place?
Today’s sandboxes are becoming the fastest and easiest way to have an overview of the threat. Ergo, Anti sandbox detection is often used to circumvent detection. Advanced malware is engineered specifically to detect when it is running in a sandbox. When that happens, the malware will avoid taking any malicious actions and evade detection. Subsequently, when the malware has been allowed to enter the network and finds itself in a real machine, it will begin its malicious behavior.
In order to avoid detection, advanced malware can alter its signature. Signatures are created by examining the internal components of an object. Skilled malware authors modify these components while preserving the object’s functionality. There are multiple transformation techniques used by malware authors and applying any of the following procedures can alter a signature:
• Code permutation
• Register renaming
• Expanding and shrinking code
• Insertion of garbage code or other constructs
As comments suggests, the subject of "malware vs. anti-virus software" as a whole is complex, and generally too broad for an answer here. However, I'd like to address what I believe may be a misconception on your part that led to your question.
In the question, you write (my emphasis):
I've been reading a lot about polymorphic and metamorphic malware and how they evade signature-based detection once they have been discovered.
and in a comment you add (also my emphasis):
I think those techniques are employed more to evade detection by AVs and IDSs once the malware has already been discovered, analyzed, and signatures extracted. I was wondering if there are any analogous techniques used specifically to prevent the malware's initial discovery so that no one would bother reporting, analyzing, or extracting signatures from it in the first place.
From my reading of the above, what I think you think happens is:
- Someone deploys a new piece of malware.
- It is discovered, analysed and "detection signatures" are extracted. AV software is now primed against the new malware.
- The malware now starts to use polymorphic measures to evade further detection.
I'm not an expert in the malware vs. anti-virus battle, but what I believe happens is more along the lines of:
- Various examples of existing malware are analysed and "detection signatures" are extracted. These can vary enormously in their complexity/subtlety, but at their simplest, these are sequences of bytes that any future malware – derived in a simplistic way from existing malware – are likely to contain.
- Someone creates a new piece of malware. To disguise its origins, and to try and evade existing "detection signatures", the writer adopts one or more polymorphic techniques.
- If "successful", the new malware won't be automatically detected by AV software: it will require manual detection. This would often only happen after the undesirable effects of the malware have triggered, and infected machines are forensically analysed.
How does malware evade initial discovery?
By obfuscating it's source code. Let's say that a tool creates a malware (in a interpreted programming language), which source is always a bit different. By mixing the code around, encrypting part of the source, different types of encoding, etc... It makes itself composed a bit different each time.
After that when the source is yet again being compiled into an executable (such is .exe), we may take advantage of additional encoding and encryption. The malware doesn't need to have the key for decrypting itself, but instead, it is using brute-force for that matter. (just an example)
This makes it hard to compare two executable and conclude that it's the same peace of software. There are additional techniques, such is deploying the source over multiple machines. Each machine doing a part of a process/operation.
Note that not every malware does "nasty" things up on the execution. And the act of decryption may raise a few alarms, but it is not a major issue.
Evading analysis of the infrastructure?
Modern malware requires a back-end, which is composed of services. Mostly APIs and graphical interfaces. There are different layers of inner network, each serving its purpuse.
Evading the analysis of a malware's infrastructure may be done by deploying critical software inside of a trusted network. A network which sits behind a fire-walls, proxies, etc... (Yes, it does requires a firewall) It's not just one executable in question, but a whole infrastructure is deployed. The end malware executable is just the infection vector. Even if it becomes easily recognizable, the organization may code (or buy) another and deploy over a existing infrastructure.
It's near impossible to get inside of a trusted network, where the back-end services are. Even if you try to hack your way into it, they will know that, and act accordingly. Maybe by deleting the instance, and erasing every clue of existence. Software in use is updated and patched. Probably with solid configuration in place. Note that a lot of different software is required to make this working. Single executable may be a footstep into the machine, but from that point, more software is deployed in order provide post exploitation. The footstep executable maybe something benign.