How effective signature-based Antivirus is, in detecting encrypted malware?
Signature-based antivirus detectors do not fare well against polymorphic or metamorphic encrypted viruses. The viruses generate a random encryption key for each new infection, so the bulk of the virus is always different. For the decryption routine itself (a fraction of the overall virus), they generate new equivalent code every time they propagate. In machine language, especially on a complex processor, there are many instructions that will accomplish identical tasks: for example, both MOV AX,0 and XOR AX,AX will achieve the result of placing a zero in the AX register. The routine can change which registers it will use - this time it uses AX for a counter, the next it uses DX. It can reorder the instructions. And it can pad the routine with a lot of randomly placed NOPs and other non-essential instructions.
All these can thwart a virus scanner, which is based on scanning for a specific sequence of bytes.
A heuristic virus scanner can look for other signs of virus activity, such as code that is writing code - that's not normally something a home or office user would do. And the NX bit can enable Data Execution Prevention (DEP), which is a CPU flag that prevents a machine from executing self-modifying code. Of course, a self-compressed executable falls under this description, so it's also not a perfect solution.