Emulators and debuggers are not the same scenario at all. Emulators run programs in isolated environment which might or might not be easily detected depending on the effort spent. Debuggers are running code in the host environment. It also could be detected using multiple techniques like starting a (second) probe debugger, look for code modification/debugger libs, exception effect analysis, hardware breakpoint detection or measuring time.

I think there is a fundamental problem with the "debugger approach": If we try to monitor the malware in the debugger -- which is certainly possible -- we are running the malicious code in the production environment using the host's libs, network and kernel. It's hardly a good idea.

Understanding the malware's inner mechanics and capabilities might need several run attempts. How does the debugger (software or human) know what steps are potentially dangerous? Sometimes it's very hard to predict.

Of course we could isolate the network access, we could copy the whole production system with checkpoints using virtual environment or separate hardware and test on that, but now we are just using the traditional approach.

If your question was whether the debugger could make things even worse (opening new attack vectors), then no, I don't think so. Classic debug process based on inserting (overwriting) a special assembly instruction (int 3 on x86 proc) in the code at the desired location, interrupting the debugged program and let the debugger control the process. Unless there is some serious vulnerability in the kernel's SIGTRAP handler, the process can not elevate privileges in this way and is running in the usual context. Having int 3 not terminating the process (which is usually the default) may present some new opportunities for delicate side channel attacks, but that's all. Because we are actually running the malware in our production environment, it can not be much worse.

TLDR: Don't run potentially malicious code in production environment. Even in a debugger.

If the question was whether the debugger could make things even worse (opening new attack vectors), then no, I don't think so. Classic debug process based on inserting (overwriting) a special assembly instruction (int 3 on x86 proc) in the code at the desired location, interrupting the debugged program and let the debugger control the process. Unless there is some serious vulnerability in the kernel's SIGTRAP handler, the process can not elevate privileges in this way and is running in the usual context. Having int 3 not terminating the process (which is usually the default) may present some new opportunities for delicate side channel attacks, but that's all. Because we are actually running the malware in our production environment, it can not be much worse.

For experimenting and calibrating ML algorithms I'd suggest a honeypot system which mimics the production one, but doesn't handle any critical information. It could be run parallel with the production system.

Additional info: Emulators and debuggers are different. Emulators run programs in isolated environment which might or might not be easily detected depending on the effort spent. Debuggers are running code in the host environment. It also could be detected using multiple (but different) techniques like starting a (second) probe debugger, look for code modification/debugger libs, exception effect analysis, hardware breakpoint detection or measuring time.