A piece of malware detects signatures of the sandbox an AV solution tries to use to fingerprint malicious behavior and pretends to be innocent. Once in the real OS environment, it then downloads executable data encoded as image data and decodes the executable bytes from the image, storing them in memory out of order, preventing any attempt at scanning the memory range for malicious code. The program then executes those instructions in the correct order, without the AV ever being able to scan them (even statically).

This scenario seems to me, with my limited experience in the field, to pose a problem for malware defense solutions.

I'd like to ask what methods are used to defeat this kind of attack, and also pose my own idea:

The kernel / os / processor (not sure which would be capable) imposes a restriction: A program must present its executable data which will become an immutable memory range that the instruction pointer cannot leave. Why is this not a good solution?

  • Firstly to execute data you need it to be executable… so it’s just a obstacle to overcome for malware. Not a barrier that stops them. And not all processors have a way to create non Executable memory. And, just having the code in non executable memory does not mean you can’t influence an application to get it from there into executable memory…
    – LvB
    Jun 1, 2021 at 22:41
  • @LvB Im proposing that the kernal make a special memory space that cannot be manipulated but can be executed and the instruction pointer is locked in that range.
    – john doe
    Jun 1, 2021 at 22:54
  • Changed read-only to "immutable" to avoid confusion with the traditional meaning of read-only memory.
    – john doe
    Jun 1, 2021 at 22:56
  • If the proc has no concept of non-exec memory it’s artificial and easily bypassed. If it does it’s just a matter of getting some other piece of the software to copy the value from the memory into exec space (think of gadgets and ROP)
    – LvB
    Jun 1, 2021 at 22:56
  • 2
    Basically, you are wanting something that would not really help but would inconvenience people. You think this because of your incomplete knowledge of assembly and procs/ OSes (so, keep learning, you are getting there in time… asking the question is a good sign). And hey, I could be wrong too.
    – LvB
    Jun 1, 2021 at 23:31

1 Answer 1


If you're proposing that the process can only present this executable data once, then it breaks an enormous range of things:

  • All software that dynamically loads libraries, be they system libraries that are different by platform version, plug-ins / mods, or simply code that keeps its memory footprint small until it knows what libraries it'll need.
  • All software that performs any type of just-in-time compilation (this rules out all modern JS engines, most Java and .NET runtimes, probably many database engines, and a bunch of other stuff that optimizes executable instructions of any sort).

Those are really important categories of software. You could accept the loss of JITs if you don't mind horrible performance costs, but think about how much of the web runs on Node now and how bad it would be if suddenly all that code was a tenth as fast?

Also, that wouldn't stop the malware anyhow, it'd just arrange the payload code and then create a new process whose entry point is that code. You could maybe stop that too, but it's tricky, then you're running into lots of other things like normal compilers.

Even just a rule that, once code is ever marked executable, it's forever non-writable, that's still going to cause problems for the entire category of debuggers (setting a breakpoint typically requires modifying the executable code, hardware breakpoints exist but they're very limited, and some other stuff debuggers do is even less practical). Also that clearly isn't enough to stop malware, because it'll just create new executable ranges at need.

Conceptually, the stuff you describe is possible today (all modern CPUs can enforce "instruction pointer can't leave [range]", that's what NX/DEP is, so it's just a matter of the OS not allowing programs to remap memory to executable or executable to writable). However, that really doesn't stop things. For example, one of the best techniques for bypassing DEP - Return-Oriented Programming (ROP) - already operates on the fact that any non-trivial program has enough "gadgets" (instructions or instruction fragments that, if you put the instruction pointer at the right place, do something useful and then jump or return to an address determined by writable stack or heap memory, said address containing the next gadget) to build a Turing machine. It might be slow and inefficient, but you can re-purpose the x86[_64] binary executable code in any non-trivial program to run literally any software, restricted only by the limitations of RAM (including stack space), CPU time, and process privileges.

There's also a thing where you seem to be focusing on trying to stop malware by writing a better antivirus and blocking ways around it. The problem is, AV is fundamentally a reactive defense - it can't do anything against novel attacks, and anything with too sensitive of heuristics will throw an unacceptable level of false positives so mostly AV just looks for known malware - and getting around AV is easy. Most malware isn't written with enough mutational capacity to avoid AV once the malware is recognized, but that's almost certainly a problem with a generalizable solution (if nothing else, malware could be written as a return-oriented program, which does nothing suspicious until it "accidentally" slips up a jump and lands in the middle of an opcode and starts doing something totally different) and so, at best, you're massively disrupting the entire software industry to just try to raise the bar for malware a little.

  • Thanks a ton. Huge help, I'm set straight. So is a aolution that guarantees the defender the ability to observe and evaluate executed code, even statically?
    – john doe
    Jun 2, 2021 at 12:48
  • Also a sidenote, not that it fixes all the reasons I was wrong, but for JIT, seeks like the JIT compiler could present the resulting executable code generated from the JS to the OS to request to append it as executable range. I'm fundamentally interested in just requiring executable code to be observable by the system for consideration before execution
    – john doe
    Jun 2, 2021 at 12:52
  • In practice, modern systems almost all use W^X, which means memory is not both writable and executable at the same time (although this does make some debugger operations slower). Thus, in practice the OS can already observe any code that a process wishes to make executable: it's the memory in any mprotect/VirtualProtect call where the mode is gaining the execute permission. Strictly enforcing W^X (or just blocking anything that violates it, without explicit user permission) would mean AV could reliably view code before execution.
    – CBHacking
    Jun 3, 2021 at 8:26
  • that's somewhat exciting - I was beginning to think maybe info sec in the context of modern OSs might be a zero sum game after all. So does that mean there exists a fundamental ability to track every instruction passed to a processor via an observation tool? Could we essentially generate a log of assembly instructions executed by any program? (without the program being able to figure out its in an emulator / debugger)
    – john doe
    Jun 5, 2021 at 22:01
  • @johndoe No, that would require hardware support that is not currently present in commodity CPUs. The reason is that, even if you enforce strict W^X and have the OS log all executable code, there's multiple ways to execute that code! X86 instructions (and the widely-used Thumb-2 variant of ARM instructions) are variable-width, and the CPU has no way to know whether the instruction pointer is aimed at the start of a "real" instruction or a fragment of one (which might overlap "real" instruction boundaries). This offset execution stream can do totally different things than the "real" code.
    – CBHacking
    Jun 6, 2021 at 5:27

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .