Recently I've started to learn more about the way antivirus and other anti malware products work, and I realized that most of them mainly use 3 different techniques to detect and stop malware:

  1. Signature-based detection
  2. Heuristic analysis
  3. Sandboxing (for the files that go through the filtering of 1. and 2.)

At first it seemed to me like a great idea, especially the sandboxing smart features that may be able to detect delayed execution, virtual machine countermeasures, encryption and more.

Then a way to bypass those detection techniques came through my mind, so I tested it and it works against every antivirus I tried and probably on any one relying on those techniques:

What if the malicious code is encrypted in an innocent looking file with a pass phrase that will be known only by the victim (who is completely unaware of it of course) and that'll be needed to decrypt the malicious code and run it at execution time?

Let's say some malicious code is encrypted using AES256 encryption function with a specific pass phrase. Then the encrypted resulting string is put into a JScript file that contains the instructions to decrypt it once the user type the pass phrase and run it through an eval() function at runtime. There is no way for the antivirus to know nor guess the pass phrase right? And no way to know there is malicious code hidden inside that JScript file. And since it'll be decrypted in memory only at runtime, without ever being written on the disk, signature-based detection and heuristics analysis will fail. Same for the sandboxing since no malicious action will be triggered without the user typing the pass phrase.

Now to make the user type the pass phrase, a simple kind of Captcha may be used, that asks the user to type the words he sees in a picture (or ASCII drawing for example ^^) in order to make sure he's not a robot. Then when the user will type it, the malicious code will be decrypted and executed.

How would one defend against this sort of attack? Perhaps an advanced antivirus using deep learning techniques might be able to detect it, but it seems to me that current antivirus programs are completely vulnerable to it.

Updated: code sample illustrating the above technique

Here is a simple implementation in JScript :

var encryptionFunction = ; // Any encryption function, e.g. AES256 from CryptoJS

var hiddenCode = ; // Write any code here, e.g. WScript.Echo('My code has been executed !');

var passphrase = "Any pass phrase"; // It'll be used as a CAPTCHA challenge-response, e.g. picture with the passphrase or ASCII drawing

var encryptedCode = encryptionFunction(hiddenCode, passphrase); // Our hidden code will be encrypted with the passphrase

var template = "var decryptionFunction = ;" + // The decryption function is defined here

               "var passphrase = InputBox('Your CAPTCHA challenge goes here');" + // CAPTCHA in an Input Box, user's input = passphrase

               "eval(decryptionFunction('" + encryptedCode + "', passphrase))"; // Decryption of the encrypted code thanks to the user's input and run at execution time through eval()

var file = ; // Create a new JScript file (or WSF) and put the template as its content

08/04/2017 - Proof of concept + Antivirus scan

I have added a proof of concept:

It's basically a simple WSF file that will display a window asking for a password, hidden in an ASCII drawing. Once the user types the correct password, the encrypted code embedded inside the file will be decrypted and executed at runtime, displaying a mere "My code has been executed !" inside a window.

As I thought, the decrypting routine wasn't considered as malicious by any antivirus on VirusTotal (0/56), here is the antivirus scan:

  • The scope of my question was more specific, expecting primarily opinion-based answers related to the technique mentioned, paving the way for new detection methods that could be used by new "non-classical" antivirus to counter those kinds of threats.
    – user143530
    Mar 31, 2017 at 0:39
  • I gave a practical, concrete example of technique and asked for opinions about practical, concrete ways to counter it. I agree the title question isn't great but I didn't find a concise way to express it. The title question could have been "What detection method could protect from the threat hereunder ?"
    – user143530
    Mar 31, 2017 at 0:48
  • 1
    @techraf - I agree with your original assessment but I have fixed some of the wording to make it more focused. At its core I believe it is a good question.
    – TTT
    Mar 31, 2017 at 4:46
  • 1
    You are essentially asking how to deal with encrypted malware where the key to the malware is somehow known to the end user but not the antivirus. Insofar I consider it a duplicate of Scan password protected email attachments for malware and the proposed measures like simply blocking the file if it looks encrypted apply here too. Apart from that: antivirus is only one part of the protection and not the single protection you should use. Mar 31, 2017 at 4:50
  • @SteffenUllrich - The file itself won't be encrypted, it'll look like an usual JS file : a mere String variable inside will store the encrypted code and there'll be an instruction to decrypt that String. I have described how to generate such a file here
    – user143530
    Mar 31, 2017 at 7:55

2 Answers 2


Good encryption looks like random data to someone without the key. Hence an antivirus program could not tell if encrypted data is malicious or not. No antivirus in the world can break through AES.

So have you found the magical silver bullet to fool all antivirus programs? No.

First of all, there needs to be some kind of bootstrapping mechanism that takes the key, decrypts the code and runs it. An email with an JS attachment is already suspicious. If the attachment decrypts a binary blob and executes it... well, that should send some red lights flashing.

So while the actual payload (the encrypted data) could not be analyzed by the antivirus, the bootstrapper could be caught by heuristics.

Second, a virus that required the user to perform some task (e.g. enter a key) is a bit unpractical. Sure, social engineering can be powerful, but this is just not as good as something that silently executes in the background with zero user interaction.

  • The key part of the technique is that the decryption key is inside the code itself, it requires no additional information, it's self-contained. And that key can only be understood by the human user (e.g., inside a CAPTCHA challenge / ASCII art) who will execute the malware. It won't have any meaning for an automated antivirus/sandbox process. But a deep learning process could make sense of it
    – user143530
    Apr 3, 2017 at 9:55
  • So the deep learning was about the CAPTCHA? I thought you meant that deep learning could help the AV to understand that the encrypted data was malicious.
    – Anders
    Apr 3, 2017 at 10:42
  • Anyway, important part is this: You need something to bootstrap it. That bootstraper could be caught with heuristics.
    – Anders
    Apr 3, 2017 at 10:42
  • Yes, the deep learning process was intended to detect the need of entering a password and how to read it like a human would. Trying to catch the bootstraper with heuristics will result in catching legitimate encryption/decryption mechanisms
    – user143530
    Apr 3, 2017 at 10:46
  • @JoëlL Edited my answer to remove the reference to deep learning. Thanks for clarifying.
    – Anders
    Apr 3, 2017 at 10:49

From my understanding this question narrows down a more general question to a very specific single example. The more general question is: how can some analysis system (antivirus etc) detect that some software is malicious if it only behaves malicious in a specific environment, with specific input data etc. This can be a password protected archive where the password is known to the user (like contained as text in the phishing mail), this can be code encrypted with a password which needs to be entered by the user, this can be the dependency on a specific OS, installed software or similar.

The answer in all these cases is: no analysis software can detect for sure that the analyzed software can be malicious. At most it might employ some heuristics which detects pattern typically associated with such behavior, like checking the specific OS and browser version or installed software, using data entered by the user for operations which might look like decryption etc. But these heuristics are not fully reliable, i.e. they might detect problems where no malicious behavior is and they will miss a lot of real problems.

Therefore such analysis can only be one part of the protection. Further parts might be to detect unusual system behavior, downloads from sites with bad reputation or only allowing whitelisted software to run.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy