Doing malware analysis I'm interested in being able to log every instruction the CPU executes for a given process. I was hoping this was perhaps possible in a standard VM environment, even if not possible in a bare-metal execution scenario. I'm interested specifically in Windows 10 for now.

To be clear, I don't mean inside a debugger or special emulator sandbox designed specifically for malware detection. That's going to be detectable by the attacker and can be bypassed. I'm thinking of an approach where we monitor the execution on our real, production machines (perhaps having all systems operate as VMs/hypervisors if necessary) and as the forbidden tools execute we analyze and detect the behavior fingerprint. This would include any dynamically allocated executable data, for example data made executable after a VirtualProtect call.

Solutions I'm considering

Processors already handle context switches as a regular part of multi-threaded architectures, for example Hyper Threading, and I guess one way to describe what I'm interested in doing is to track the state of one of those contexts as the VM simulates it.

Considering a general arithmetic addition operation A = B+1. The instruction is stored in the instruction register and the program counter is incremented. A and B are read from memory and are stored in registers R1, R2 respectively. In this case, B+1 is calculated and written in R1 as the final answer. This operation as there are sequential reads and writes and there's no waits for function calls used, hence no context switch/wait takes place in this case. - source

Perhaps I would need to hijack the VM's implementation of this process to track which process is currently in filling the register state and insert some code in the VM's hyperthreading emulation code to store that state somewhere else each time it updates?

Perhaps that's overkill. Perhaps no VMs actually virtualize systems on such a low level to make this possible. I'm still learning about the lower level working of VMs. I would expect such low-level emulation to cause a significant performance hit, especially if we're recording register states between each instruction - a massive performance hit probably.


Can this be achieved with existing virtual machine tools, or why not? What features should I look to in VM software or Windows OS documentation? Or rather, will I almost certainly have to make low-level modifications to a VM product to create a custom solution for doing this? Or perhaps would a new kind of VM have to be built to essentially do this kind of instruction processing virtualization?

  • 3
    What does knowing the CPU operations get you? It's the I/O where the danger lies, and that is easier to monitor.
    – schroeder
    Commented Jun 9, 2021 at 19:16
  • @schroeder Currently detection of behavior is evaded by polymorphic tools, example EMOTET. Consider the obfuscation of steps A and B to convert input 1 to output 2 into steps A, A1, A2, A3, B, B1, B2, B3, to convert input 1 to output 2. Now let's say we decompile that original code and re-obfuscate it not only that one way but with a hundred obfuscators and thousands of variations. I think machine learning can be leveraged to look at the changes in register / memory / storage states and create a fingerprint by noticing the key pattern of [input 1, A, B, output 2] occurring in all versions.
    – J.Todd
    Commented Jun 9, 2021 at 19:52
  • @schroeder (2/2) If we just look at the data loading into memory to be executed, we see what the program might be able to do, but based on many factors the program might do a wide number of things, and of course there's ROP and such. I think the key to really winning the cat and mouse game in behavior analysis is to be able to observe what the program does, and a lot of that behavior isn't necessarily visible in I/O, if I'm a clever attacker I'm keeping as much of my activity as possible tucked away in the registers. The I/O doesn't expose as much of the behavior as the internals of the CPU
    – J.Todd
    Commented Jun 9, 2021 at 20:45
  • Anything you do in software that runs on the same CPU could be detectable by an attacker for some of the same reasons a debugger is detectable. For example, a debugger introduces real-time delays that are not present without a debugger. So too will any logging software running on the same CPU. And if you are logging all the instructions, you will be introducing a lot of delays, which is detectable. Also, your CPU executes billions of operations a second--will you be logging them all? Maybe a custom hardware solution is best? But seems like a lot of work...
    – hft
    Commented Jun 9, 2021 at 21:24
  • 2
    You're trying to put in so much time and effort into replicating a debugger. If you say that it doesn't matter if the attacker knows they are being watched, just go with a debugger. Or a hardware debugger if you wish
    – nobody
    Commented Jun 10, 2021 at 8:53

2 Answers 2


Yes, some VMs can do that, but only by using emulation, never virtualization.

It's absolutely not possible to do on bare metal without JTAG (which, due to its limited bandwidth, reduces performance massively), nor is it possible to do in a VM while using hardware-accelerated virtualization like VT-x or AMD-v because only certain instructions can be configured to trap and return control to the hypervisor. You can't trap, for example, the test instruction when running in a virtual machine, so it will execute without you being able to record or influence it.

The only solution is to use a VM which emulates instructions, like QEMU in softmmu mode or Bochs. Note that this will also come with a massive performance hit, and the malware you're analyzing will likely realize that it's running in a VM, because there are going to be differences between the VM and real hardware. It's also potentially insecure, and QEMU does not consider bugs in softmmu to be security issues. However, you will be able to record and even adjust individual instructions. You'll be able to do everything you could do with GDB, but on the level of the entire OS.

Obviously malware would have an easy time knowing that it's running under some sort of debugger or emulator, but since you can manipulate individual instructions, registers, and memory, you can force the malware to think it is running on bare metal. Of course, this requires an understanding of the malware and how it operates, which itself requires a working knowledge of reverse engineering. Being able to step through individual instructions and modify the execution context at will makes that easier, because you're not limited to changing the malware's environment and running it blindly.

  • I think as far as winning the behavior analysis ca and mouse game, we have to analyze what the program actually does (emulating everything it could do is interesting but with a large program there are so many paths to emulate it gets computationally massive) and apply machine learning. (just looking at sys calls or I/O isn't good enough, either) There are many, many papers on this topic which I'm reading. (1/3)
    – J.Todd
    Commented Jun 24, 2021 at 12:20
  • Program Synthesis I believe will initially lead the way in this field's near future. The Monte Carlo family of algorithms is a beautiful and powerful tool. I'm researching applying LSTM neural networks to this approach. But that leaves us with the question of how we record the program's instructions. Tracing the activity on your endpoints by running a debugger trace on all your threads is not only big overhead, but the attacker will know. I like the idea of the attacker not knowing. (2/3)
    – J.Todd
    Commented Jun 24, 2021 at 12:23
  • So ultimately I think a specialized Intel processor with either a version of XDP (Intel's JTAG expansion) with many more busses with a separate port for each physical core, or perhaps a completely new system separate from JTAG with a faster protocol is the long term end game for endpoint cyber security. Ultimately to externally monitor a modern CPU we need ~1GB/s bandwidth per physical core to keep up with the fetch decode execute cycle.
    – J.Todd
    Commented Jun 24, 2021 at 12:30
  • 1
    @J.Todd I'm sure Intel has in-house devices which can do that, but it's unlikely to be possible without the CPU being half-open. Such devices likely cost Intel millions to manufacture.
    – forest
    Commented Jun 24, 2021 at 23:40
  • 1
    @J.Todd: Do note that even if you could get the data copied into the Hyperthread peer in real time (which I agree, Hyperthreading is closely coupled enough to minimize needed hardware changes, but without those changes you don't get even partway), you still can't get it out in anything approaching real-time. Also your 1GBps estimate was far too low, even 2.5 years ago. L1 Cache bandwidth per core has been over 100GBps for a while, and you are proposing to capture both instruction and data streams, so double that again.
    – Ben Voigt
    Commented Dec 18, 2023 at 19:43

In Windows, there's Time travel debugging. Despite the name, according to Bhasali.ppt it logs instructions by JIT-translating them.

It does incur a considerable performance penalty, and as such is probably not suitable for production use.

  • 1
    Also note that debugging was rejected by the OP ...
    – schroeder
    Commented Dec 18, 2023 at 15:52

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .