This question already has an answer here:

How effective signature-based Antivirus is, in detecting encrypted malware?

marked as duplicate by John Deters, Xander, Lucas Kauffman, TildalWave, Adi Jan 6 '14 at 21:11

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • It depends on the antivirus but they are mostly not very effective. Tool-kits like Metasploit have pretty advanced encryption / obfuscation tools for their exploits that will generally bypass most of the existing software. – Awake Zoldiek Jan 6 '14 at 13:39
  • they are not effective. They (almost only) catch encrypted malware only if they have the encyrpted signature. – cengizUzun Jan 6 '14 at 15:52

Signature-based antivirus detectors do not fare well against polymorphic or metamorphic encrypted viruses. The viruses generate a random encryption key for each new infection, so the bulk of the virus is always different. For the decryption routine itself (a fraction of the overall virus), they generate new equivalent code every time they propagate. In machine language, especially on a complex processor, there are many instructions that will accomplish identical tasks: for example, both MOV AX,0 and XOR AX,AX will achieve the result of placing a zero in the AX register. The routine can change which registers it will use - this time it uses AX for a counter, the next it uses DX. It can reorder the instructions. And it can pad the routine with a lot of randomly placed NOPs and other non-essential instructions.

All these can thwart a virus scanner, which is based on scanning for a specific sequence of bytes.

A heuristic virus scanner can look for other signs of virus activity, such as code that is writing code - that's not normally something a home or office user would do. And the NX bit can enable Data Execution Prevention (DEP), which is a CPU flag that prevents a machine from executing self-modifying code. Of course, a self-compressed executable falls under this description, so it's also not a perfect solution.

Not the answer you're looking for? Browse other questions tagged or ask your own question.