2

Doing malware analysis I'm interested in being able to log every instruction the CPU executes for a given process. I was hoping this was perhaps possible in a standard VM environment, even if not possible in a bare-metal execution scenario. I'm interested specifically in Windows 10 for now.

To be clear, I don't mean inside a debugger or special emulator sandbox designed specifically for malware detection. That's going to be detectable by the attacker and can be bypassed. I'm thinking of an approach where we monitor the execution on our real, production machines (perhaps having all systems operate as VMs/hypervisors if necessary) and as the forbidden tools execute we analyze and detect the behavior fingerprint. This would include any dynamically allocated executable data, for example data made executable after a VirtualProtect call.

Solutions I'm considering

Processors already handle context switches as a regular part of multi-threaded architectures, for example Hyper Threading, and I guess one way to describe what I'm interested in doing is to track the state of one of those contexts as the VM simulates it.

Considering a general arithmetic addition operation A = B+1. The instruction is stored in the instruction register and the program counter is incremented. A and B are read from memory and are stored in registers R1, R2 respectively. In this case, B+1 is calculated and written in R1 as the final answer. This operation as there are sequential reads and writes and there's no waits for function calls used, hence no context switch/wait takes place in this case. - source

Perhaps I would need to hijack the VM's implementation of this process to track which process is currently in filling the register state and insert some code in the VM's hyperthreading emulation code to store that state somewhere else each time it updates?

Perhaps that's overkill. Perhaps no VMs actually virtualize systems on such a low level to make this possible. I'm still learning about the lower level working of VMs. I would expect such low-level emulation to cause a significant performance hit, especially if we're recording register states between each instruction - a massive performance hit probably.

Question

Can this be achieved with existing virtual machine tools, or why not? What features should I look to in VM software or Windows OS documentation? Or rather, will I almost certainly have to make low-level modifications to a VM product to create a custom solution for doing this? Or perhaps would a new kind of VM have to be built to essentially do this kind of instruction processing virtualization?

9
  • 2
    What does knowing the CPU operations get you? It's the I/O where the danger lies, and that is easier to monitor.
    – schroeder
    Jun 9 '21 at 19:16
  • @schroeder Currently detection of behavior is evaded by polymorphic tools, example EMOTET. Consider the obfuscation of steps A and B to convert input 1 to output 2 into steps A, A1, A2, A3, B, B1, B2, B3, to convert input 1 to output 2. Now let's say we decompile that original code and re-obfuscate it not only that one way but with a hundred obfuscators and thousands of variations. I think machine learning can be leveraged to look at the changes in register / memory / storage states and create a fingerprint by noticing the key pattern of [input 1, A, B, output 2] occurring in all versions.
    – J.Todd
    Jun 9 '21 at 19:52
  • @schroeder (2/2) If we just look at the data loading into memory to be executed, we see what the program might be able to do, but based on many factors the program might do a wide number of things, and of course there's ROP and such. I think the key to really winning the cat and mouse game in behavior analysis is to be able to observe what the program does, and a lot of that behavior isn't necessarily visible in I/O, if I'm a clever attacker I'm keeping as much of my activity as possible tucked away in the registers. The I/O doesn't expose as much of the behavior as the internals of the CPU
    – J.Todd
    Jun 9 '21 at 20:45
  • Anything you do in software that runs on the same CPU could be detectable by an attacker for some of the same reasons a debugger is detectable. For example, a debugger introduces real-time delays that are not present without a debugger. So too will any logging software running on the same CPU. And if you are logging all the instructions, you will be introducing a lot of delays, which is detectable. Also, your CPU executes billions of operations a second--will you be logging them all? Maybe a custom hardware solution is best? But seems like a lot of work...
    – hft
    Jun 9 '21 at 21:24
  • As a simpler alternative, maybe just log API calls or syscalls? It sort of gives you similar information since you can figure out the instructions for any given function, then if you know what functions were called with what arguments you known what instructions were executed.
    – hft
    Jun 9 '21 at 21:28
2

Yes, some VMs can do that, but only by using emulation, never virtualization.

It's absolutely not possible to do on bare metal without JTAG (which, due to its limited bandwidth, reduces performance massively), nor is it possible to do in a VM while using hardware-accelerated virtualization like VT-x or AMD-v because only certain instructions can be configured to trap and return control to the hypervisor. You can't trap, for example, the test instruction when running in a virtual machine, so it will execute without you being able to record or influence it.

The only solution is to use a VM which emulates instructions, like QEMU in softmmu mode or Bochs. Note that this will also come with a massive performance hit, and the malware you're analyzing will likely realize that it's running in a VM, because there are going to be differences between the VM and real hardware. However, you will be able to record and even adjust individual instructions. You'll be able to do everything you could do with GDB, but on the level of the entire OS.

Obviously malware would have an easy time knowing that it's running under some sort of debugger or emulator, but since you can manipulate individual instructions, registers, and memory, you can force the malware to think it is running on bare metal. Of course, this requires an understanding of the malware and how it operates, which itself requires a working knowledge of reverse engineering. Being able to step through individual instructions and modify the execution context at will makes that easier, because you're not limited to changing the malware's environment and running it blindly.

6
  • I think as far as winning the behavior analysis ca and mouse game, we have to analyze what the program actually does (emulating everything it could do is interesting but with a large program there are so many paths to emulate it gets computationally massive) and apply machine learning. (just looking at sys calls or I/O isn't good enough, either) There are many, many papers on this topic which I'm reading. (1/3)
    – J.Todd
    Jun 24 '21 at 12:20
  • Program Synthesis I believe will initially lead the way in this field's near future. The Monte Carlo family of algorithms is a beautiful and powerful tool. I'm researching applying LSTM neural networks to this approach. But that leaves us with the question of how we record the program's instructions. Tracing the activity on your endpoints by running a debugger trace on all your threads is not only big overhead, but the attacker will know. I like the idea of the attacker not knowing. (2/3)
    – J.Todd
    Jun 24 '21 at 12:23
  • So ultimately I think a specialized Intel processor with either a version of XDP (Intel's JTAG expansion) with many more busses with a separate port for each physical core, or perhaps a completely new system separate from JTAG with a faster protocol is the long term end game for endpoint cyber security. Ultimately to externally monitor a modern CPU we need ~1GB/s bandwidth per physical core to keep up with the fetch decode execute cycle.
    – J.Todd
    Jun 24 '21 at 12:30
  • @J.Todd I'm sure Intel has in-house devices which can do that, but it's unlikely to be possible without the CPU being half-open. Such devices likely cost Intel millions to manufacture.
    – forest
    Jun 24 '21 at 23:40
  • It just dawned on me that Intel already sort of has the infrastructure to achieve this with existing silicon. Hyperthreading provides at most 30% performance increase, for a security oriented chip that does what I'm asking for the security process could run on the twin set of registers for each each physical CPU. Add a new instruction for those twin hyperthread processes to trap its twin's instructions and do some processing. The two sets of registers are in such physical proximity that an instruction to copy the most recently executed instruction from the twin's tread could be very fast.(1/2)
    – J.Todd
    Jun 29 '21 at 10:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.