How to defend against a virus that hides the payload by encrypting it?

Question

Recently I've started to learn more about the way antivirus and other anti malware products work, and I realized that most of them mainly use 3 different techniques to detect and stop malware:

1. Signature-based detection
2. Heuristic analysis
3. Sandboxing (for the files that go through the filtering of 1. and 2.)

At first it seemed to me like a great idea, especially the sandboxing smart features that may be able to detect delayed execution, virtual machine countermeasures, encryption and more.

Then a way to bypass those detection techniques came through my mind, so I tested it and it works against every antivirus I tried and probably on any one relying on those techniques:

What if the malicious code is encrypted in an innocent looking file with a pass phrase that will be known only by the victim (who is completely unaware of it of course) and that'll be needed to decrypt the malicious code and run it at execution time?

Let's say some malicious code is encrypted using AES256 encryption function with a specific pass phrase. Then the encrypted resulting string is put into a JScript file that contains the instructions to decrypt it once the user type the pass phrase and run it through an eval() function at runtime. There is no way for the antivirus to know nor guess the pass phrase right? And no way to know there is malicious code hidden inside that JScript file. And since it'll be decrypted in memory only at runtime, without ever being written on the disk, signature-based detection and heuristics analysis will fail. Same for the sandboxing since no malicious action will be triggered without the user typing the pass phrase.

Now to make the user type the pass phrase, a simple kind of Captcha may be used, that asks the user to type the words he sees in a picture (or ASCII drawing for example ^^) in order to make sure he's not a robot. Then when the user will type it, the malicious code will be decrypted and executed.

How would one defend against this sort of attack? Perhaps an advanced antivirus using deep learning techniques might be able to detect it, but it seems to me that current antivirus programs are completely vulnerable to it.

Updated: code sample illustrating the above technique

Here is a simple implementation in JScript :

var encryptionFunction = ; // Any encryption function, e.g. AES256 from CryptoJS

var hiddenCode = ; // Write any code here, e.g. WScript.Echo('My code has been executed !');

var passphrase = "Any pass phrase"; // It'll be used as a CAPTCHA challenge-response, e.g. picture with the passphrase or ASCII drawing

var encryptedCode = encryptionFunction(hiddenCode, passphrase); // Our hidden code will be encrypted with the passphrase

var template = "var decryptionFunction = ;" + // The decryption function is defined here

               "var passphrase = InputBox('Your CAPTCHA challenge goes here');" + // CAPTCHA in an Input Box, user's input = passphrase

               "eval(decryptionFunction('" + encryptedCode + "', passphrase))"; // Decryption of the encrypted code thanks to the user's input and run at execution time through eval()

var file = ; // Create a new JScript file (or WSF) and put the template as its content

08/04/2017 - Proof of concept + Antivirus scan

I have added a proof of concept:
https://gist.github.com/Joel-L-G/6cd8aede4126e71f361aeb90eecf4999

It's basically a simple WSF file that will display a window asking for a password, hidden in an ASCII drawing. Once the user types the correct password, the encrypted code embedded inside the file will be decrypted and executed at runtime, displaying a mere "My code has been executed !" inside a window.

As I thought, the decrypting routine wasn't considered as malicious by any antivirus on VirusTotal (0/56), here is the antivirus scan:
https://www.virustotal.com/en/file/a7d453fa3aa23d10144ce13b2a7a94120f664368f35a5ca32dc731c608f14c0e/analysis/

Answer 1

Good encryption looks like random data to someone without the key. Hence an antivirus program could not tell if encrypted data is malicious or not. No antivirus in the world can break through AES.

So have you found the magical silver bullet to fool all antivirus programs? No.

First of all, there needs to be some kind of bootstrapping mechanism that takes the key, decrypts the code and runs it. An email with an JS attachment is already suspicious. If the attachment decrypts a binary blob and executes it... well, that should send some red lights flashing.

So while the actual payload (the encrypted data) could not be analyzed by the antivirus, the bootstrapper could be caught by heuristics.

Second, a virus that required the user to perform some task (e.g. enter a key) is a bit unpractical. Sure, social engineering can be powerful, but this is just not as good as something that silently executes in the background with zero user interaction.

Answer 2

From my understanding this question narrows down a more general question to a very specific single example. The more general question is: how can some analysis system (antivirus etc) detect that some software is malicious if it only behaves malicious in a specific environment, with specific input data etc. This can be a password protected archive where the password is known to the user (like contained as text in the phishing mail), this can be code encrypted with a password which needs to be entered by the user, this can be the dependency on a specific OS, installed software or similar.

The answer in all these cases is: no analysis software can detect for sure that the analyzed software can be malicious. At most it might employ some heuristics which detects pattern typically associated with such behavior, like checking the specific OS and browser version or installed software, using data entered by the user for operations which might look like decryption etc. But these heuristics are not fully reliable, i.e. they might detect problems where no malicious behavior is and they will miss a lot of real problems.

Therefore such analysis can only be one part of the protection. Further parts might be to detect unusual system behavior, downloads from sites with bad reputation or only allowing whitelisted software to run.