Monitoring system calls (in a reliable and secure way)

Question

It there a reliable method of “monitoring” system calls under Linux?

There is strace for example to monitor system calls and signals. Is there a way for a process to dodge out of strace? If yes, is there another reliable, secure method of "monitoring" system calls (and, maybe receiving signals), that a process cannot escape from (assuming a proper Linux implementation)?

Answer 1

If yes, Is there another reliable, secure method of "monitoring" system calls (and, maybe receiving signals), that process can not break (assuming proper Linux implementation) ?

To re-iterate in a slightly different way what D.W. has already said, ptrace is a system call that strace, gdb and the like make to monitor a processes' actions. There are two problems with this approach:

1. As you probably know, hooking system calls is a favourite technique of rootkit authors. It's entirely possible to replace ptrace, provide you the output of another process or some other such malfeasance.

2. Processes are not always written to submit willingly to debuggers. You might like to read this challenge set (win32 focused - see the very first entry and keep reading for making it difficult) from an appsec company (I have no links with them). Whilst focused on the IsDebuggerPresent() mechanism, similar solutions exist for ptrace. If you want to see this in the wild and have skype installed on a Linux box, try debugging it.

   Repeating those techniques here, there are two clear anti-ptrace mechanisms:

   if (ptrace(PTRACE_TRACEME, 0, 0, 0) < 0) {
       printf("being ptraced\n");
       exit(1);
   }
   

   This method relies on the fact a process cannot be traced twice. If you can't ptrace yourself, you're being ptraced.

   struct timespec spec;
   
   signal(SIGALRM, SIG_IGN);
   alarm(1);
   spec.tv_sec = 2;
   spec.tv_nsec = 0;
   if (nanosleep(&spec, NULL) < 0) {
       /* EINTR */
       printf("being ptraced\n");
       exit(1);
   }
   

   To explain this one, have a look at nanosleep() and read the original article. In simple terms, nanosleep() is a non-restartable system call and will return early when a signal is handled by the process. This particular process will, when not being debugged, just not handle that particular signal and so will not be woken up. However, a ptraced process will handle it, causing nanosleep to return early. Another example where this happens is the select() system call.

Ultimately, you can mitigate the effects of 1 by ensuring the integrity of your system prior to starting and applying sufficient security measures and an appropriately configured kernel.

What can you do reliably about 2? Not a lot without modifying the original binary code, since any technique for debugging is going to introduce observable inconsistencies or implementation issues somewhere.

tl;dr ptrace will help you if the target process wasn't written with debuggers in mind.

Answer 2

On Linux, you can reliably monitor a selection of system calls or file accesses with the audit subsystem. Make sure the auditd daemon is running, then configure what you want to log with auditctl. Each logged operation is recorded in /var/log/audit/audit.log (on typical configurations).

You'll find simple examples of auditctl usage on this site, on Server Fault, and on Unix Stack Exchange.

strace or associated programs using ptrace are reliable ways of monitoring system calls, but I would be wary of using them on a malicious program. I couldn't tell you how off the top of my head, but it should be possible for a monitored program to make the right ptrace calls to evade monitoring.

Note that a malicious program could spawn a process that is not audited and can execute code that won't be logged. For example, it could use mmap to write to a file without the file contents ever appearing as the arguments of system calls, make this file executable and spawn a process executing it. The spawned process can typically break the process tree with something like ssh localhost. If you audit all the processes executed by a given user (as opposed to only a single process and its descendants), you'll be able to log everything.

Answer 3

The Linux Audit Framework supports syscall monitoring - I believe it is what you are looking for.

Answer 4

Yes. strace is a reasonable way to monitor system calls and their arguments, as long as the process being monitored is not malicious. If the process being monitored is malicious and was written to evade strace, I expect that it can do so. strace was not written as a security tool, and I can hypothesize several ways that the process might defeat it. See, e.g., Robert Watson, Exploiting Concurrency Vulnerabilities in System Call Wrappers or Tal Garfinkel, Traps and Pitfalls: Practical Problems in System Call Interposition Based Security Tools.

If you are worried about malicious code, you'll want to use a sandbox that was designed for security, rather than a tool like strace that wasn't designed for security. The general approach to building such a sandbox is to use system call interposition both to contain the monitored process, and to monitor its actions. One portable method is to use ptrace, though this can introduce a non-trivial performance overhead as it forces a context switch on every system call. On Solaris, you can use /proc; /proc lets you specify the subset of system calls that you are interested in wrapping, which lets you achieve better performance at the cost of compatibility.

Take a look at Plash, Systrace, and Subterfugue, to see some worked systems that use these sorts of methods. Also look at Chrome's sandbox, which uses a variety of mechanisms (including seccomp on Linux).