Has exploitation been demonstrated against the fundamental constructs of the debugging process?

Question

I'm curious to know if an attacker can fundamentally exploit the debugging process.

I'm not asking if specific debugging tools have been exploitable, surely some have, but rather whether the process of debugging - any and perhaps every debugging tool - is vulnerable to being lied to or used as an attack surface.

If the answer is "it depends on which OS we're talking about", let's focus on the Windows debugging API. And if the answer is that the way Windows does it isn't reliable or secure but it fundamentally could be, then that's what I'd like to know.

This relates to research I'm doing on the problem space of modern malware using polymorphism, emulator evasion, and obfuscation to defeat signature recognition.

It doesn't matter if the attacker knows it's running in a debugger

I'm fully aware that an attacker can detect they're running in a modified environment (whitepaper by Blackthorne et al. based on emulator detection but debuggers create the same scenario) (video) and use evasion strategies to hide the malicious behavior they might otherwise demonstrate on a target machine. However, I'm interested in the usage of a debugger (Qira, a "timeless" debugger by Google's Project Zero is of interest, although not decidedly so) to track the state of a process on the target machine so I can experiment with applying machine learning to develop signatures of polymorphic, obfuscated, and evasive programs.

Because we're watching a process on the actual target rather than within an EV emulator sandbox, it doesn't matter that the attacker knows they're being watched by a debugger. The attacker has to execute the malicious behavior path within their code in order for the payload to accomplish its goal.

My goal is detection of a signature - perhaps interruption of a detected process, but that's not mandatory. I'm looking at this problem space from the perspective of making a reliable way to create a signature for recognizing the polymorphic obfuscated payload for the purpose of identifying and responding to the threat rapidly, even if it already did the bad thing.

Performance isn't of interest

(So long as it's still feasible to run basic office programs performantly)

I realize running a program in a debugger will cause a significant performance hit. That's an accepted trade-off of the research I'm interested in. We're assuming the defender is willing to pay higher hardware costs to run programs. If certain computationally expensive programs can't afford this overhead, the security measure wouldn't apply to certain hosts running those programs. Such hosts may be isolated in a specific network segment with that factor in mind.

My question is this:

Is there a fundamental flaw in the reliability of the way debugging can be accomplished? Is the way debugging happens vulnerable at the processor, kernel, or OS level? Is it possible for a properly designed debugger to reliably watch the state of a program in a production environment as it executes?

Answer 1

Debuggers are fundamentally ahead in this game. A perfectly written debugger will always be able to perfectly simulate a runtime environment where a perfectly written malware could never detect. Real life debuggers are complicated software that can have many vulnerabilities that may allow a particularly sophisticated malware to detect that it's running in a debugger, but in theory debugger have the inherent field advantage.

However, one complicating situation here is that in the real world, poking holes in a real world software/hardware with bugs and writing malware to exploit those is a much easier task than debugging an unknown program and understanding whether it contains malicious behaviour. So in practice, malware do have lots of practical advantage.

Answer 2

If the question was whether the debugger could make things even worse (opening new attack vectors), then no, I don't think so. Classic debug process based on inserting (overwriting) a special assembly instruction (int 3 on x86 proc) in the code at the desired location, interrupting the debugged program and let the debugger control the process. Unless there is some serious vulnerability in the kernel's SIGTRAP handler, the process can not elevate privileges in this way and is running in the usual context. Having int 3 not terminating the process (which is usually the default) may present some new opportunities for delicate side channel attacks, but that's all. Because we are actually running the malware in our production environment, it can not be much worse.

For experimenting and calibrating ML algorithms I'd suggest a honeypot system which mimics the production one, but doesn't handle any critical information. It could be run parallel with the production system.

Additional info: Emulators and debuggers are different. Emulators run programs in isolated environment which might or might not be easily detected depending on the effort spent. Debuggers are running code in the host environment. It also could be detected using multiple (but different) techniques like starting a (second) probe debugger, look for code modification/debugger libs, exception effect analysis, hardware breakpoint detection or measuring time.