Is there a virus that attempted to foil generic decryption techniques? How did it do so?
I need to know the virus name, because I tried searching and I am unable to locate any virus that did that, and the mechanism of the virus is not defined either.
Thanks for the link, it is an interesting question indeed. I'm not a virus expert, but I know from previous reading I know that from the document you mention, I have a problem with the sentence "Inside this virtual computer, program files execute as if running on a real computer" which merely sounds like marketing simplification to my hear.
Indeed, the issue with this system is that if the anti-virus would have to implement a full fledged virtual machine to run the virus in an undetectable sandboxed environment really implement a full computer environment, then it would be really too resources intensive. Therefore, the anti-virus designer have to find a compromise: which system call should be implemented, which should be just stub, what kind of values should be returned, etc.
Starting from that, the virus designer will rely on these assumption to try to detect:
- When the virus code is run from within the anti-virus sandbox (trivial example: call some system function several times and compare the results which should be different on a real system, and are always the same due to anti-virus' sandbox stub), in this case it will not trigger the decryption algorithm and remain undetectable,
- When the virus code is run on the real system, in this case the virus will trigger its load.
And the next logical steps are just an arms race. Anti-virus vendors try either to fool or recognize the virus' sandbox detection routines, and the virus designers try to find different alternative ways to detect the sandbox, and so on...
Edit: Interestingly enough, the document goes on and acknowledges the speed issue affecting the ideal "as if running on a real computer" model ("the key problem with generic decryption is speed", page 8). It then replaces this ideal model with an heuristic one, concluding with the "marketing version" of the arms race I mentioned: "heuristics demand continual research and updating".